Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-1378

FIWARE.Request.Lab.SophiaAntipolis.Sophia Antipolis.Image instance with error not terminating. C4I Region

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-LAB-HELP
    • Labels:
      None
    • HD-Node:
      SophiaAntipolis

      Description

      Hello,

      I cant seem to be able to terminate image instances that have an error status.
      They show Task as deleting but it has no effect.

      Browser : Chrome Version 38.0.2125.104
      OS : Linux version 3.13.0-37-generic (buildd@roseapple) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #64-Ubuntu SMP Mon Sep 22 21:30:01 UTC 2014

      _____________________________
      Abbas AHMAD
      Ingènieur Informatique - Génie Logiciel
      FEMTO-ST - Université de Franche-Comté
      abbas.ahmad@edu.univ-fcomte.fr
      +33 (0)6.47.56.85.94

      _______________________________________________
      Fiware-lab-help mailing list
      Fiware-lab-help@lists.fi-ware.org
      https://lists.fi-ware.org/listinfo/fiware-lab-help

        Activity

        Hide
        fw.ext.user FW External User added a comment -

        Dear Abbas,

        Errored Vms cnannot be removed directly.
        They have to be reset first (force state to ACTIVE) by using command line tools.
        Then (mostly) they can be removed.
        But this does not work always.

        Best Regards, Bernd

        ============
        Bernd Bochow
        Next Generation Network Infrastructures
        Fraunhofer Institute for Open Communication Systems (FOKUS)
        Kaiserin-Augusta-Allee 31, D-10589 Berlin
        e-mail: bernd.bochow@fokus.fraunhofer.de, bernd.bochow@ieee.org
        phone: +49 30 3463-7238
        fax: +49 30 3463-997238

        From: abbas ahmad <abbas.ahmad@edu.univ-fcomte.fr<abbas.ahmad@edu.univ-fcomte.fr>>
        Date: Monday 17 November 2014 09:47
        To: "fiware-lab-help@lists.fi-ware.org<fiware-lab-help@lists.fi-ware.org>" <fiware-lab-help@lists.fi-ware.org<fiware-lab-help@lists.fi-ware.org>>
        Subject: [Fiware-lab-help] Image instance with error not terminating. C4I Region

        Hello,

        I cant seem to be able to terminate image instances that have an error status.
        They show Task as deleting but it has no effect.

        Browser : Chrome Version 38.0.2125.104
        OS : Linux version 3.13.0-37-generic (buildd@roseapple) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #64-Ubuntu SMP Mon Sep 22 21:30:01 UTC 2014

        Show
        fw.ext.user FW External User added a comment - Dear Abbas, Errored Vms cnannot be removed directly. They have to be reset first (force state to ACTIVE) by using command line tools. Then (mostly) they can be removed. But this does not work always. Best Regards, Bernd ============ Bernd Bochow Next Generation Network Infrastructures Fraunhofer Institute for Open Communication Systems (FOKUS) Kaiserin-Augusta-Allee 31, D-10589 Berlin e-mail: bernd.bochow@fokus.fraunhofer.de, bernd.bochow@ieee.org phone: +49 30 3463-7238 fax: +49 30 3463-997238 From: abbas ahmad <abbas.ahmad@edu.univ-fcomte.fr< abbas.ahmad@edu.univ-fcomte.fr >> Date: Monday 17 November 2014 09:47 To: "fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >" <fiware-lab-help@lists.fi-ware.org< fiware-lab-help@lists.fi-ware.org >> Subject: [Fiware-lab-help] Image instance with error not terminating. C4I Region Hello, I cant seem to be able to terminate image instances that have an error status. They show Task as deleting but it has no effect. Browser : Chrome Version 38.0.2125.104 OS : Linux version 3.13.0-37-generic (buildd@roseapple) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #64-Ubuntu SMP Mon Sep 22 21:30:01 UTC 2014
        Hide
        abbas.ahmad Abbas Ahmad added a comment -

        Thanks for the fast reply. But i cant seem to reboot the instance as it is in error state.

        As for vnc i cant access the instance over it for now. waiting to know how to configure a private network address to allocate a public ip address on it and bind the all with the instance.

        Show
        abbas.ahmad Abbas Ahmad added a comment - Thanks for the fast reply. But i cant seem to reboot the instance as it is in error state. As for vnc i cant access the instance over it for now. waiting to know how to configure a private network address to allocate a public ip address on it and bind the all with the instance.
        Hide
        BerlinSupportTeam Berlin Node Helpdesk added a comment -

        Could you please provide your tenant id - it shows up after longing into the portal in the light blue area on the bottom. Many Thanks.

        Show
        BerlinSupportTeam Berlin Node Helpdesk added a comment - Could you please provide your tenant id - it shows up after longing into the portal in the light blue area on the bottom. Many Thanks.
        Hide
        abbas.ahmad Abbas Ahmad added a comment -

        Info: Connected to project ahmad (ID 00000000000000000000000000006355)

        Show
        abbas.ahmad Abbas Ahmad added a comment - Info: Connected to project ahmad (ID 00000000000000000000000000006355)
        Hide
        BerlinSupportTeam Berlin Node Helpdesk added a comment -

        Many thanks.

        I cannot delete the instances too. All three of them errored due to a quantum problem (in a get_request to poll the VM status showing up as an authentication failure).
        It seems that the nova host and the computing node are out of sync which can happen after a nova host reboot.
        Usually, this corrects itself within 24 hours when the nova host and the computing nodes resync their rabbitmq state.
        The VMs are probably already deleted but the nova host doesn't get the correct status return from the computing node.

        The error is

        {"message": "TypeError", "code": 500, "details": "unsupported operand type(s) for +: 'NoneType' and 'str' | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 224, in decorated_function | | | return function(self, context, *args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1416, in terminate_instance | | | do_terminate_instance(instance, bdms) | | | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 242, in inner | | | retval = f(*args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1408, in do_terminate_instance | | | reservations=reservations) | | | File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 85, in inner | | | rv = f(*args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1371, in _delete_instance | | | project_id=project_id) | | | File \"/usr/lib/python2.7/contextlib.py\", line 24, in __exit__ | | | self.gen.next() | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1344, in _delete_instance | | | self._shutdown_instance(context, instance, bdms) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1262, in _shutdown_instance | | | network_info = self._get_instance_nw_info(context, instance) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 697, in _get_instance_nw_info | | | instance, conductor_api=self.conductor_api) | | | File \"/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py\", line 368, in get_instance_nw_info | | | result = self._get_instance_nw_info(context, instance, networks) | | | File \"/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py\", line 376, in _get_instance_nw_info | | | nw_info = self._build_network_info_model(context, instance, networks) | | | File \"/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py\", line 795, in _build_network_info_model | | | data = client.list_ports(**search_opts) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 107, in with_params | | | ret = self.function(instance, *args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 255, in list_ports | | | **_params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 996, in list | | | for r in self._pagination(collection, path, **params): | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 1009, in _pagination | | | res = self.get(path, params=params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 982, in get | | | headers=headers, params=params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 967, in retry_request | | | headers=headers, params=params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 904, in do_request | | | resp, replybody = self.httpclient.do_request(action, method, body=body) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/client.py\", line 151, in do_request | | | self.authenticate() | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/client.py\", line 180, in authenticate | | | token_url = self.auth_url + \"/tokens\" | | | ", "created": "2014-11-17T08:18:28Z"}

        |

        Show
        BerlinSupportTeam Berlin Node Helpdesk added a comment - Many thanks. I cannot delete the instances too. All three of them errored due to a quantum problem (in a get_request to poll the VM status showing up as an authentication failure). It seems that the nova host and the computing node are out of sync which can happen after a nova host reboot. Usually, this corrects itself within 24 hours when the nova host and the computing nodes resync their rabbitmq state. The VMs are probably already deleted but the nova host doesn't get the correct status return from the computing node. The error is {"message": "TypeError", "code": 500, "details": "unsupported operand type(s) for +: 'NoneType' and 'str' | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 224, in decorated_function | | | return function(self, context, *args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1416, in terminate_instance | | | do_terminate_instance(instance, bdms) | | | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 242, in inner | | | retval = f(*args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1408, in do_terminate_instance | | | reservations=reservations) | | | File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 85, in inner | | | rv = f(*args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1371, in _delete_instance | | | project_id=project_id) | | | File \"/usr/lib/python2.7/contextlib.py\", line 24, in __exit__ | | | self.gen.next() | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1344, in _delete_instance | | | self._shutdown_instance(context, instance, bdms) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1262, in _shutdown_instance | | | network_info = self._get_instance_nw_info(context, instance) | | | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 697, in _get_instance_nw_info | | | instance, conductor_api=self.conductor_api) | | | File \"/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py\", line 368, in get_instance_nw_info | | | result = self._get_instance_nw_info(context, instance, networks) | | | File \"/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py\", line 376, in _get_instance_nw_info | | | nw_info = self._build_network_info_model(context, instance, networks) | | | File \"/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py\", line 795, in _build_network_info_model | | | data = client.list_ports(**search_opts) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 107, in with_params | | | ret = self.function(instance, *args, **kwargs) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 255, in list_ports | | | **_params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 996, in list | | | for r in self._pagination(collection, path, **params): | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 1009, in _pagination | | | res = self.get(path, params=params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 982, in get | | | headers=headers, params=params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 967, in retry_request | | | headers=headers, params=params) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py\", line 904, in do_request | | | resp, replybody = self.httpclient.do_request(action, method, body=body) | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/client.py\", line 151, in do_request | | | self.authenticate() | | | File \"/usr/lib/python2.7/dist-packages/quantumclient/client.py\", line 180, in authenticate | | | token_url = self.auth_url + \"/tokens\" | | | ", "created": "2014-11-17T08:18:28Z"} |
        Hide
        BerlinSupportTeam Berlin Node Helpdesk added a comment -

        I tried again to delete the instances after 2h hours but failed.
        I tried to launch an instance but failed with the same error (in creating the instance).
        There seems to be a configuration flaw.
        I would like to ask C4I to check in particular if the quantum can reach the keystone for authentication as seen in the error message (check if endpoints can be reached).

        in nova.conf:
        [DEFAULT]
        os_region_name=C4I
        quantum_region_name=C4I

        I cannot do that from the Berlin node side.
        Many Thanks!

        Show
        BerlinSupportTeam Berlin Node Helpdesk added a comment - I tried again to delete the instances after 2h hours but failed. I tried to launch an instance but failed with the same error (in creating the instance). There seems to be a configuration flaw. I would like to ask C4I to check in particular if the quantum can reach the keystone for authentication as seen in the error message (check if endpoints can be reached). in nova.conf: [DEFAULT] os_region_name=C4I quantum_region_name=C4I I cannot do that from the Berlin node side. Many Thanks!
        Hide
        BerlinSupportTeam Berlin Node Helpdesk added a comment -

        I cannot resolve this - could you please help?

        Show
        BerlinSupportTeam Berlin Node Helpdesk added a comment - I cannot resolve this - could you please help?
        Hide
        BerlinSupportTeam Berlin Node Helpdesk added a comment -

        I cannot find any VM related to your tenant ID (00000000000000000000000000006355) on the Berlin node.

        Show
        BerlinSupportTeam Berlin Node Helpdesk added a comment - I cannot find any VM related to your tenant ID (00000000000000000000000000006355) on the Berlin node.
        Hide
        bastien Bastien Putegnat (Inactive) added a comment -

        Hello,

        sorry for the late reply.
        We had issues on our last support email. We changed recently. ( See attachment).
        The image instance issues has been fixed. Our nodes was in maintenance mode.
        we apologize for the inconvenience.

        Bastien.P.

        Show
        bastien Bastien Putegnat (Inactive) added a comment - Hello, sorry for the late reply. We had issues on our last support email. We changed recently. ( See attachment). The image instance issues has been fixed. Our nodes was in maintenance mode. we apologize for the inconvenience. Bastien.P.

          People

          • Assignee:
            Com4Innov Sophia Antipolis Node Helpdesk
            Reporter:
            abbas.ahmad Abbas Ahmad
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: