Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-3866

FIWARE.Request.Lab.Spain.instance unaccesible.

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 2021
    • Component/s: FIWARE-LAB-HELP
    • Labels:
      None

      Description

      Hi,

      I'm Ignacio Gomez from FINODEX_035. Today while I was working on my
      instance, suddenly I can't access to it through ssh. It requests me for
      a password when it never does. I do not know what it is happening.
      The instance is running on : 130.206.113.111 and I can manage it from
      fiware lab. Instance id is: 3a77c3ea-1b66-4dfd-ac05-67082eaa610d
      I think I have everything properly working because I access to the
      instance in the same way several hours before. I also tried from
      different computers (linux and windows).
      It is quite urgent we are running two pilots (from FINODEX and
      SmartAgriFood) from the same instance.

      Regards
      Ignacio Gomez
      SensoWave

      _______________________________________________
      Fiware-lab-help mailing list
      Fiware-lab-help@lists.fi-ware.org
      https://lists.fi-ware.org/listinfo/fiware-lab-help

      [Created via e-mail received from: Ignacio Maqueda <imaqueda@sensowave.com>]

        Activity

        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        Fixed changing the permission of /root directory

        Show
        jmperibanez Jose Maria Peribañez added a comment - Fixed changing the permission of /root directory
        Hide
        fw.ext.user FW External User added a comment -

        Thanks for the information. I'll change the root password asap.

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Thanks for the information. I'll change the root password asap. Regards Ignacio
        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        Great!

        Well, the problem was because /root permission was changed, I suppose you or other person of your team modify this permission by accident, perhaps running a script. Be careful with the operations under root account

        Another suggestion, put a really hard password (it is important that is a random string, enough long, with uppercase and lowercase letters and numbers; I've observed in the logs that people has tried to enter in the VM) under root or better, create another account with root access via sudo, to make it possible to enter using the console if something goes really wrong.

        Regards,

        Chema.

        Show
        jmperibanez Jose Maria Peribañez added a comment - Great! Well, the problem was because /root permission was changed, I suppose you or other person of your team modify this permission by accident, perhaps running a script. Be careful with the operations under root account Another suggestion, put a really hard password (it is important that is a random string, enough long, with uppercase and lowercase letters and numbers; I've observed in the logs that people has tried to enter in the VM) under root or better, create another account with root access via sudo, to make it possible to enter using the console if something goes really wrong. Regards, Chema.
        Hide
        fw.ext.user FW External User added a comment -

        Hi Chema,

        That is great. It works again. Many thanks.What might happen to not do
        it again?

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi Chema, That is great. It works again. Many thanks.What might happen to not do it again? Regards Ignacio
        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        Hi,

        Please, try to connect again using the ssh key to the root account. I've reset the /root permission.

        About creating another VM in the same network. The network you have to select is node-int-net-01. This is a big network (192.168.192.0/18, i.e. 192.168.192.1 -> 192.168.255.254)

        Chema.

        Show
        jmperibanez Jose Maria Peribañez added a comment - Hi, Please, try to connect again using the ssh key to the root account. I've reset the /root permission. About creating another VM in the same network. The network you have to select is node-int-net-01. This is a big network (192.168.192.0/18, i.e. 192.168.192.1 -> 192.168.255.254) Chema.
        Hide
        fw.ext.user FW External User added a comment -

        Hi Chema,

        I was trying to create another instance to get data from context broker
        but they are not in the same network. How can I do to create another
        instance in the same subnet that the orion instance (192.168.204.148)?
        Basically I need to create another instance to setup PROTON GE to
        monitor data incoming to Context Broker. However I need that both are in
        the same subnet to enter to the Proton instance by ssh from context
        instance that it is the instance with public ip.

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi Chema, I was trying to create another instance to get data from context broker but they are not in the same network. How can I do to create another instance in the same subnet that the orion instance (192.168.204.148)? Basically I need to create another instance to setup PROTON GE to monitor data incoming to Context Broker. However I need that both are in the same subnet to enter to the Proton instance by ssh from context instance that it is the instance with public ip. Regards Ignacio
        Hide
        fw.ext.user FW External User added a comment -

        Hi Chema,

        I can access to the instance by using localadmin account however I don't
        know if I'm going to be able to setting up Proton.
        As I said before, if it is possible to turn off the instance and later
        turn it on again without losing anything, go ahead. If we may lose data,
        we have to think about it.

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi Chema, I can access to the instance by using localadmin account however I don't know if I'm going to be able to setting up Proton. As I said before, if it is possible to turn off the instance and later turn it on again without losing anything, go ahead. If we may lose data, we have to think about it. Regards Ignacio
        Hide
        fw.ext.user FW External User added a comment -

        Hi Chema,

        What do you mean by halting? Stop it and restart it later. Will I loose
        the work in the instance? I'll try later to access it with localadmin
        account and I'll comment you

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi Chema, What do you mean by halting? Stop it and restart it later. Will I loose the work in the instance? I'll try later to access it with localadmin account and I'll comment you Regards Ignacio
        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        Hi again,

        I've found the problem probably, but unfortunately it cannot be fixed without halting the VM. I cannot mount the disk of a running VM as read-write without a serious risk of corrupting it.

        The ssh connection probably fails because of a bad permission in /root (permission is rwxrwx-w-), this is something that somebody must has changed accidentally in the morning of 23 Aug. Indeed, the last success access is at 10.33, but at 13.01 the error is "Authentication refused: bad ownership or modes for directory /root"

        Probably you can log in the VM using your public key to localadmin account instead of root, but the problem is to get root privileges.

        In the future the VMs will have a support account to fix problems like this, but nowadays I'm afraid that there is not a "hot fix" to this issue

        Tomorrow morning I'll comment to my colleagues about allocating an extra IP address. But first try to log to the localadmin account, if you need a redirection using a port above 1024 and the port is not in use in the host, a temporal workaround is to create an SSH redirection.

        Regards,

        Chema.

        Show
        jmperibanez Jose Maria Peribañez added a comment - Hi again, I've found the problem probably, but unfortunately it cannot be fixed without halting the VM. I cannot mount the disk of a running VM as read-write without a serious risk of corrupting it. The ssh connection probably fails because of a bad permission in /root (permission is rwxrwx-w-), this is something that somebody must has changed accidentally in the morning of 23 Aug. Indeed, the last success access is at 10.33, but at 13.01 the error is "Authentication refused: bad ownership or modes for directory /root" Probably you can log in the VM using your public key to localadmin account instead of root, but the problem is to get root privileges. In the future the VMs will have a support account to fix problems like this, but nowadays I'm afraid that there is not a "hot fix" to this issue Tomorrow morning I'll comment to my colleagues about allocating an extra IP address. But first try to log to the localadmin account, if you need a redirection using a port above 1024 and the port is not in use in the host, a temporal workaround is to create an SSH redirection. Regards, Chema.
        Hide
        fw.ext.user FW External User added a comment -

        Hi again,

        Another option to, at least, setting up CEP enabler is to have another
        IP address. In this way I can redirect incoming information to this
        other machine where we will setup the CEP enabler. Is it possible?

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi again, Another option to, at least, setting up CEP enabler is to have another IP address. In this way I can redirect incoming information to this other machine where we will setup the CEP enabler. Is it possible? Regards Ignacio
        Hide
        fw.ext.user FW External User added a comment -

        Hi Chema,

        Thanks for your effort. What options do I have because I have a big
        problem? Do you want the pem file to access to my instance? Would it help?
        I mean I have to access to the virtual machine to setup CEP enabler and
        to setup security on Orion access because I have to add it to my pilots
        according with my proposal. However, at the same time, I can not remove
        the instance because I have pilots running using that instance. Maybe I
        can prepare an email to the coordinators of my accelerator programs and
        tell them about this issue.

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi Chema, Thanks for your effort. What options do I have because I have a big problem? Do you want the pem file to access to my instance? Would it help? I mean I have to access to the virtual machine to setup CEP enabler and to setup security on Orion access because I have to add it to my pilots according with my proposal. However, at the same time, I can not remove the instance because I have pilots running using that instance. Maybe I can prepare an email to the coordinators of my accelerator programs and tell them about this issue. Regards Ignacio
        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        Hi Ignacio,

        I cannot reproduce the problem. I've created a new VM and I connect correctly using a RSA key (I had a problem using a DSS key, but different than yours).

        I tried to upgrade packages, because the only logical explanation after discarding a network redirection problem is a software upgrade, but now the VM is a CentOS 6.7 and it works without problems...

        I've spend several hours trying to reproduce the problem, but I don't know what other things to test.

        It's clear that the problem is inside the VM.

        Show
        jmperibanez Jose Maria Peribañez added a comment - Hi Ignacio, I cannot reproduce the problem. I've created a new VM and I connect correctly using a RSA key (I had a problem using a DSS key, but different than yours). I tried to upgrade packages, because the only logical explanation after discarding a network redirection problem is a software upgrade, but now the VM is a CentOS 6.7 and it works without problems... I've spend several hours trying to reproduce the problem, but I don't know what other things to test. It's clear that the problem is inside the VM.
        Hide
        fw.ext.user FW External User added a comment -

        Ok Chema,

        Thank you so much. I hope for your news. At least, machine is still
        working (I have running context broker for the fiware pilots).

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Ok Chema, Thank you so much. I hope for your news. At least, machine is still working (I have running context broker for the fiware pilots). Regards Ignacio
        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        Hi Ignacio,

        I am trying to analyse your problem deploying another VM, but without success for the moment

        Regards,

        Chema.

        Show
        jmperibanez Jose Maria Peribañez added a comment - Hi Ignacio, I am trying to analyse your problem deploying another VM, but without success for the moment Regards, Chema.
        Hide
        fw.ext.user FW External User added a comment -

        Hi Chema,

        I'm trying to connect to 130.206.113.111 with the root user. I copy
        below the response from ssh:

        ssh -v -i sensowave.pem root@130.206.113.111
        OpenSSH_6.0p1 Debian-4+deb7u1, OpenSSL 1.0.1e 11 Feb 2013
        debug1: Reading configuration data /etc/ssh/ssh_config
        debug1: /etc/ssh/ssh_config line 19: Applying options for *
        debug1: Connecting to 130.206.113.111 [130.206.113.111] port 22.
        debug1: Connection established.
        debug1: identity file sensowave.pem type -1
        debug1: identity file sensowave.pem-cert type -1
        debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
        debug1: match: OpenSSH_5.3 pat OpenSSH_5*
        debug1: Enabling compatibility mode for protocol 2.0
        debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u1
        debug1: SSH2_MSG_KEXINIT sent
        debug1: SSH2_MSG_KEXINIT received
        debug1: kex: server->client aes128-ctr hmac-md5 none
        debug1: kex: client->server aes128-ctr hmac-md5 none
        debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
        debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
        debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
        debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
        debug1: Server host key: RSA 86:5d:7e:d8:93:2b:62:74:f8:9c:3f:f7:01:71:c0:cf
        debug1: Host '130.206.113.111' is known and matches the RSA host key.
        debug1: Found key in /home/pi/.ssh/known_hosts:1
        debug1: ssh_rsa_verify: signature correct
        debug1: SSH2_MSG_NEWKEYS sent
        debug1: expecting SSH2_MSG_NEWKEYS
        debug1: SSH2_MSG_NEWKEYS received
        debug1: Roaming not allowed by server
        debug1: SSH2_MSG_SERVICE_REQUEST sent
        debug1: SSH2_MSG_SERVICE_ACCEPT received
        debug1: Authentications that can continue:
        publickey,gssapi-keyex,gssapi-with-mi c,password
        debug1: Next authentication method: gssapi-keyex
        debug1: No valid Key exchange context
        debug1: Next authentication method: gssapi-with-mic
        debug1: Unspecified GSS failure. Minor code may provide more information
        Cannot determine realm for numeric host address

        debug1: Unspecified GSS failure. Minor code may provide more information
        Cannot determine realm for numeric host address

        debug1: Unspecified GSS failure. Minor code may provide more information

        debug1: Unspecified GSS failure. Minor code may provide more information
        Cannot determine realm for numeric host address

        debug1: Next authentication method: publickey
        debug1: Trying private key: sensowave.pem
        debug1: read PEM private key done: type RSA
        debug1: Authentications that can continue:
        publickey,gssapi-keyex,gssapi-with-mi c,password
        debug1: Next authentication method: password

        I always work with the same pem file. Could you help?
        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi Chema, I'm trying to connect to 130.206.113.111 with the root user. I copy below the response from ssh: ssh -v -i sensowave.pem root@130.206.113.111 OpenSSH_6.0p1 Debian-4+deb7u1, OpenSSL 1.0.1e 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to 130.206.113.111 [130.206.113.111] port 22. debug1: Connection established. debug1: identity file sensowave.pem type -1 debug1: identity file sensowave.pem-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u1 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Server host key: RSA 86:5d:7e:d8:93:2b:62:74:f8:9c:3f:f7:01:71:c0:cf debug1: Host '130.206.113.111' is known and matches the RSA host key. debug1: Found key in /home/pi/.ssh/known_hosts:1 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mi c,password debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information Cannot determine realm for numeric host address debug1: Unspecified GSS failure. Minor code may provide more information Cannot determine realm for numeric host address debug1: Unspecified GSS failure. Minor code may provide more information debug1: Unspecified GSS failure. Minor code may provide more information Cannot determine realm for numeric host address debug1: Next authentication method: publickey debug1: Trying private key: sensowave.pem debug1: read PEM private key done: type RSA debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mi c,password debug1: Next authentication method: password I always work with the same pem file. Could you help? Regards Ignacio
        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        Goog morning Ignacio,

        I'm Chema, from FiWare. I'm assigned to your incident.

        The only logical explanation that I found is that the SSH is redirected to another VM because a network problem (but the only situation where this may be possible is with a dark configuration created several month ago, that is not your case). I've checked that all is right. Anyway, have you seen a message similar to:
        @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
        @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
        @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
        IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
        Someone could be eavesdropping on you right now (man-in-the-middle attack)!
        It is also possible that a host key has just been changed.

        What user account are you trying to connect? the root account?

        Regards,

        Chema

        Show
        jmperibanez Jose Maria Peribañez added a comment - Goog morning Ignacio, I'm Chema, from FiWare. I'm assigned to your incident. The only logical explanation that I found is that the SSH is redirected to another VM because a network problem (but the only situation where this may be possible is with a dark configuration created several month ago, that is not your case). I've checked that all is right. Anyway, have you seen a message similar to: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. What user account are you trying to connect? the root account? Regards, Chema
        Hide
        jmperibanez Jose Maria Peribañez added a comment -

        I've checked if there is a network problem that redirects to a wrong VM, but all seems OK: there are not old distributed rourters (it is using the shared networks), the floating IP is assigned to the right VM, the iptables entries exists in the rourter namespace...

        Show
        jmperibanez Jose Maria Peribañez added a comment - I've checked if there is a network problem that redirects to a wrong VM, but all seems OK: there are not old distributed rourters (it is using the shared networks), the floating IP is assigned to the right VM, the iptables entries exists in the rourter namespace...
        Hide
        fw.ext.user FW External User added a comment -

        Hi Manuel,

        I'm working with Spain2 node.

        Regards
        Ignacio

        Show
        fw.ext.user FW External User added a comment - Hi Manuel, I'm working with Spain2 node. Regards Ignacio
        Hide
        mev Manuel Escriche added a comment -

        Hi Ignacio,

        What node are you working with? Spanish?

        Knowing it will allow me forwarding your request to the right team.

        Thanks.
        Manuel

        Show
        mev Manuel Escriche added a comment - Hi Ignacio, What node are you working with? Spanish? Knowing it will allow me forwarding your request to the right team. Thanks. Manuel

          People

          • Assignee:
            jmperibanez Jose Maria Peribañez
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: