Modern versions of Ansible are equipped with a quiver of Network CLI modules that can be used to talk to network devices, e.g. routers, that do not support ssh and python, Ansible’s default method of communication and manipulation of remote hosts. These modules have names such as sros_command, junos_command, and openvswitch_db. You can find the complete list here.

Under the covers, Ansible uses the python paramiko package to connect with these various network devices. Unfortunately, paramiko error handling in the Ansible network modules is lacking. Often, we end up with a catch-all error of the following form:

2018-10-27 15:42:48,890 p=17607 u=root |  fatal: [vsc2.mgmt.vnspoc.net -> localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to open shell. Please see:https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell"
}

We have seen this error caused by many things.

Paramiko version

We have found that some of the unable to open shell errors have been caused by using the wrong version of paramiko. In our testing, paramiko versions 2.2.1 and 2.4.1 work well. You can check your version with:

pip list | grep paramiko

And install the proper version with:

pip install paramiko==2.4.1

Socket handling

As a performance enhancement, Ansible tries to keep open the socket used to connect to your network device. By not repeatedly opening and closing the socket they achieve better performance. We have found that some versions of Ansible do not manage this properly. We have tested Ansible version 2.4.0. It has shown itself to be reliable in this regard. We will be upgrading to 2.7 soon. Fingers crossed…

You can check your version of Ansible with:

ansible --version

And install the proper version with:

pip install ansible==2.4.0.0

You can test if this is your problem by manually deleting the socket file and re-trying. The socket files are located here:

/root/.ansible/pc/

Other packages

We are starting to see that the unable to open shell error is coming more frequently due to paramiko dependency packages being of the wrong version. The only way to determine this is to follow the advice in the error message and set ANSIBLE_DEBUG=true in your environment. This will produce extraordinarily verbose output. Most of that output isn’t useful, but when there is a package mismatch it can be the only way to get to the bottom of it. In one recent case, we found this output in the ansible.log file after enabling debugging:

2018-12-13 11:57:41,783 paramiko.transport Unknown exception: cannot import name certificate_transparency
2018-12-13 11:57:41,785 paramiko.transport Traceback (most recent call last):
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 1925, in run
2018-12-13 11:57:41,785 paramiko.transport     self.kex_engine.parse_next(ptype, m)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/kex_gex.py", line 91, in parse_next
2018-12-13 11:57:41,785 paramiko.transport     return self._parse_kexdh_gex_reply(m)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/kex_gex.py", line 263, in _parse_kexdh_gex_reply
2018-12-13 11:57:41,785 paramiko.transport     self.transport._verify_key(host_key, sig)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 1747, in _verify_key
2018-12-13 11:57:41,785 paramiko.transport     key = self._key_info[self.host_key_type](Message(host_key))
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 62, in __init__
2018-12-13 11:57:41,786 paramiko.transport     ).public_key(default_backend())
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.hazmat.backends.openssl.backend import backend
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.hazmat.backends.openssl.backend import backend
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 16, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography import utils, x509
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/x509/__init__.py", line 7, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.x509 import certificate_transparency
2018-12-13 11:57:41,786 paramiko.transport ImportError: cannot import name certificate_transparency
2018-12-13 11:57:41,786 paramiko.transport 
2018-12-13 11:57:41,788 p=9657 u=root |  connecting to host 10.10.62.117 returned an error
2018-12-13 11:57:41,788 p=9657 u=root |  cannot import name certificate_transparency

 

It’s pretty clear that this is a paramiko error. And all that text boils down to this error:

cannot import name certificate_transparency

A quick search of the Internet for that error shows that when this particular error is encountered the python cryptography library is often out of date. The recommended fix is to update the package:

sudo pip install –upgrade cryptography

What is outlined here is a specific fix for a specific occurrence. There is a general rule illustrated by this specific example. That rule is that paramiko errors are not handled well by paramiko and Ansible. When the generic “unable to open shell” appears, it seems the problem could be caused by a missing or out-of-date python package paramiko relies on. In such a case, enable ANSIBLE_DEBUG and parse the ansible.log.