pipdeptree

I was chasing a dependency issue when executing Python code. After much searching and several blind alleys in that search, evidence started to mount that the problem was related to a package dependency not being properly satisfied.

I started using pip show and other commands to piece together the evidence I needed to pinpoint the error. Then I stumbled upon a python tool that does the work for me: pipdeptree.

pip install pipdeptree

Then execute on a package:

[root@jen-master ~]# pipdeptree -p ansible
ansible==2.4.0.0
- cryptography [required: Any, installed: 2.3.1]
- asn1crypto [required: >=0.21.0, installed: 0.24.0]
- cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- enum34 [required: Any, installed: 1.1.6]
- idna [required: >=2.1, installed: 2.7]
- ipaddress [required: Any, installed: 1.0.22]
- six [required: >=1.4.1, installed: 1.11.0]
- jinja2 [required: Any, installed: 2.10]
- MarkupSafe [required: >=0.23, installed: 1.1.0]
- paramiko [required: Any, installed: 2.4.1]
- bcrypt [required: >=3.1.3, installed: 3.1.4]
- cffi [required: >=1.1, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- six [required: >=1.4.1, installed: 1.11.0]
- cryptography [required: >=1.5, installed: 2.3.1]
- asn1crypto [required: >=0.21.0, installed: 0.24.0]
- cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- enum34 [required: Any, installed: 1.1.6]
- idna [required: >=2.1, installed: 2.7]
- ipaddress [required: Any, installed: 1.0.22]
- six [required: >=1.4.1, installed: 1.11.0]
- pyasn1 [required: >=0.1.7, installed: 0.4.4]
- pynacl [required: >=1.0.1, installed: 1.3.0]
- cffi [required: >=1.4.1, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- six [required: Any, installed: 1.11.0]
- PyYAML [required: Any, installed: 3.13]
- setuptools [required: Any, installed: 0.9.8]

As you can see, a recursive list of every dependency along with version requirements. Very helpful!

Advertisements

Unable to open shell error

Modern versions of Ansible are equipped with a quiver of Network CLI modules that can be used to talk to network devices, e.g. routers, that do not support ssh and python, Ansible’s default method of communication and manipulation of remote hosts. These modules have names such as sros_command, junos_command, and openvswitch_db. You can find the complete list here.

Under the covers, Ansible uses the python paramiko package to connect with these various network devices. Unfortunately, paramiko error handling in the Ansible network modules is lacking. Often, we end up with a catch-all error of the following form:

2018-10-27 15:42:48,890 p=17607 u=root |  fatal: [vsc2.mgmt.vnspoc.net -> localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to open shell. Please see:https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell"
}

We have seen this error caused by many things.

Paramiko version

We have found that some of the unable to open shell errors have been caused by using the wrong version of paramiko. In our testing, paramiko versions 2.2.1 and 2.4.1 work well. You can check your version with:

pip list | grep paramiko

And install the proper version with:

pip install paramiko==2.4.1

Socket handling

As a performance enhancement, Ansible tries to keep open the socket used to connect to your network device. By not repeatedly opening and closing the socket they achieve better performance. We have found that some versions of Ansible do not manage this properly. We have tested Ansible version 2.4.0. It has shown itself to be reliable in this regard. We will be upgrading to 2.7 soon. Fingers crossed…

You can check your version of Ansible with:

ansible --version

And install the proper version with:

pip install ansible==2.4.0.0

You can test if this is your problem by manually deleting the socket file and re-trying. The socket files are located here:

/root/.ansible/pc/

Other packages

We are starting to see that the unable to open shell error is coming more frequently due to paramiko dependency packages being of the wrong version. The only way to determine this is to follow the advice in the error message and set ANSIBLE_DEBUG=true in your environment. This will produce extraordinarily verbose output. Most of that output isn’t useful, but when there is a package mismatch it can be the only way to get to the bottom of it. In one recent case, we found this output in the ansible.log file after enabling debugging:

2018-12-13 11:57:41,783 paramiko.transport Unknown exception: cannot import name certificate_transparency
2018-12-13 11:57:41,785 paramiko.transport Traceback (most recent call last):
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 1925, in run
2018-12-13 11:57:41,785 paramiko.transport     self.kex_engine.parse_next(ptype, m)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/kex_gex.py", line 91, in parse_next
2018-12-13 11:57:41,785 paramiko.transport     return self._parse_kexdh_gex_reply(m)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/kex_gex.py", line 263, in _parse_kexdh_gex_reply
2018-12-13 11:57:41,785 paramiko.transport     self.transport._verify_key(host_key, sig)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 1747, in _verify_key
2018-12-13 11:57:41,785 paramiko.transport     key = self._key_info[self.host_key_type](Message(host_key))
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 62, in __init__
2018-12-13 11:57:41,786 paramiko.transport     ).public_key(default_backend())
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.hazmat.backends.openssl.backend import backend
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.hazmat.backends.openssl.backend import backend
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 16, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography import utils, x509
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/x509/__init__.py", line 7, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.x509 import certificate_transparency
2018-12-13 11:57:41,786 paramiko.transport ImportError: cannot import name certificate_transparency
2018-12-13 11:57:41,786 paramiko.transport 
2018-12-13 11:57:41,788 p=9657 u=root |  connecting to host 10.10.62.117 returned an error
2018-12-13 11:57:41,788 p=9657 u=root |  cannot import name certificate_transparency

 

It’s pretty clear that this is a paramiko error. And all that text boils down to this error:

cannot import name certificate_transparency

A quick search of the Internet for that error shows that when this particular error is encountered the python cryptography library is often out of date. The recommended fix is to update the package:

sudo pip install –upgrade cryptography

What is outlined here is a specific fix for a specific occurrence. There is a general rule illustrated by this specific example. That rule is that paramiko errors are not handled well by paramiko and Ansible. When the generic “unable to open shell” appears, it seems the problem could be caused by a missing or out-of-date python package paramiko relies on. In such a case, enable ANSIBLE_DEBUG and parse the ansible.log.

Why not JavaScript *Without* a Framework?

For months now I have been thinking about the design of a web front end for a collection services. There are *SO MANY CHOICES* out there that it’s sometimes difficult to turn off the incoming streams and make a choice. The exercise of thinking about a design often starts with thinking about frameworks. Code frameworks, primarily written for Javascript with a few for Python, do a bunch of work for you. But there’s a price. Frameworks are often large and regularly force you to implement things in one and only one way. Frameworks are not compatible with one another, so a choice locks you into a heavy investment. Should we use NodeJS? React? Flask? The list is seemingly endless.

Yesterday, I stumbled upon a podcast that takes an alternate approach. The guest on the show, Chris Ferdinandi, asserts that the conversation need not start with a framework. He acknowledges the fact that it can be overwhelming to thinking about frameworks. In some cases, it’s better to just use JavaScript directly. The code is smaller. There’s no framework lock-in. And JavaScript has come a long way from the state it was in several years ago that led to the proliferation of frameworks for it. To help people figure out what to do, Ferdinandi created a website, Go Make Things, where we can find small guides to help us learn to do things *without* resorting to frameworks.

The podcast uses quite a bit of jargon. I didn’t understand it all. But I did find the exposure to another way of thinking to be very helpful. I plan to check out Ferdinandi’s website.

Go Make things:

https://gomakethings.com/

Hanselminutes podcast:

https://www.hanselminutes.com/598/maybe-just-use-vanilla-javascript-with-chris-ferdinandi

AnsibleFest 2017, Network Modules in 2.3, and Ansible connection type

I had the opportunity to attend AnsibleFest 2017 in San Francisco. The sessions I attended were of high quality. It was well worth the cost of attendance.

One of the cool things the Ansible folks setup were “Ask an Expert” sessions. Attendees could sign up for a 15-minute appoitment to discuss whatever issues have been troubling you. I signed up for a session with a network-module author to talk about my recent struggles running network modules using Ansible 2.3.

In Ansible 2.3, network modules require connection: local. When I ran my existing playbooks using Ansible 2.3, I saw errors of the form:

invalid connection specified, expected connection=local, got smart

I spent a bunch of time online before AnsibleFest trying to find out why this was happening. I didn’t understand connection: local, and the documentation was vague. The expert at AnsibleFest set me straight. The explanation was so simple that I had a solid “duh!” moment.

connection: local simply means that the module being invoked will be run on the Ansible node. That’s it. The default, connection: remote, pushes the module’s Python code to the inventory host and executes it there. Starting in Ansible version 2.3, network modules enforce connection: local because they operate against inventory hosts that usually don’t have Python installed, e.g. SROS.

I’ve now adopted the following best practice:

  • Configure my inventory to set connection type for localhost to ‘local’
    • [local_host]
      localhost ansible_connection=local
  • Delegate tasks that I want to run on the Ansible host to localhost
    • delegate_to: localhost

 

Ansible map() not available on el6

If you are running Ansible playbooks on an el6 machine and you run across an error like this:

2017-05-24 10:21:47,585 p=8525 u=root |  fatal: [localhost]: FAILED! =>
{"failed": true, "msg": "The conditional check '( myvsds is defined and 
( myvsds | map(attribute='target_server_type') | list | issuperset([\"kvm\"]) 
or myvsds | map(attribute='target_server_type') | list | issuperset([\"heat\"])
) ) or ( myvcins is defined and ( myvcins | map(attribute='target_server_type')
| list | issuperset([\"kvm\"]) or myvcins | map(attribute='target_server_type')
| list | issuperset([\"heat\"]) ) )' failed. The error was: template
error while templating string: no filter named 'map'. String: {% if (
myvsds is defined and ( myvsds | map(attribute='target_server_type') | list |
issuperset([\"kvm\"]) or myvsds | map(attribute='target_server_type') | list |
issuperset([\"heat\"]) ) ) or ( myvcins is defined and ( myvcins |
map(attribute='target_server_type') | list | issuperset([\"kvm\"]) or myvcins |
map(attribute='target_server_type') | list | issuperset([\"heat\"]) ) ) %} True
{% else %} False {% endif %}\n\nThe error appears to have been in 
'/metro-2.1.1/roles/build/tasks/get_paths.yml': line 8, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending
line appears to be:\n\n  - block: # QCOW2\n    - name: Find name of VSD QCOW2
File\n      ^ here\n"}

Note the text in BOLD. The problem is caused by the fact that Ansible as of version 2.1 depends on the map() filter implementation from the package python-jinja2. map() was introduced into python-jinja2 starting with python-jinja2 version 2.7. The base python-jinja2 version for el6 is 2.2, thus creating the error, above.

This means that Ansible using map() must be running el7 on the Ansible host.

Ansible [WARNING]: The loop variable ‘item’ is already in use.

I made a simple change to an existing Ansible playbook. I used the include_role command to invoke another role. Since I was calling the role on a list of hosts that I had dynamically discovered at runtime, I used with_items to make the call iterate over the list.

Not good. I saw the following warning and error:

[WARNING]: The loop variable 'item' is already in use. You should set
the `loop_var` value in the `loop_control` option for the task to 
something else to avoid variable collisions and unexpected behavior.

fatal: [localhost]: FAILED! => {
 "failed": true,
 "msg": "The conditional check ''u16' not in item|json_query('Name')
failed. The error was: error while evaluating conditional ('u16' not 
in item|json_query('Name')): 'item' is undefined... }

After a bit of searching and reading docs, I figured out how to fix. But the docs and examples were not straightforward. I hope you will find a better explanation herein.

First, Ansible (I’m using 2.2.1) doesn’t handle nested with_items loops properly. There’s something special about the way item is handled such that using item in nested loops causes one of the expected values to be overwritten.

My outer loop:

- name: Use ci-destroy to clean unused VMs
  include_role:
    name: ci-destroy
  with_items:
    - "{{ my_vm_list }}"

ci-destroy is a role that we use to garbage collect VMs from test failures in our CI environment. Before this task, the code gathers a list of the orphan VMs in the environment. The ci-destroy role is called on each one.

The ci-destroy role is my inner loop. It contains, among other things:

- name: Remove several entries from /etc/hosts file
  lineinfile:
    dest: /etc/hosts
    line: "{{ item }}"
    state: absent
  with_items: "{{ line_list }}"

With the outer loop and the inner loop using {{ item }}, Ansible had a problem. WARNING and the ERROR, as shown above.

The fix? Use loop_control to specify a non-default variable name for the inner item variable name. In my case:

- name: Remove several entries from /etc/hosts file
  lineinfile:
    dest: /etc/hosts
    line: "{{ line_item }}"
    state: absent
  with_items: "{{ line_list }}"
  loop_control:
    loop_var: line_item

The changed lines are shown in red. Basically, I changed the inner loop such that it used line_item instead of item. Worked like a charm.

Ansible dependencies via meta

I ran across an interesting feature of Ansible this week. A co-worker said that an upstream change to an open-source project he had been working on broke our installation code. The author had moved the invocation of a role from the main playbook for a role into roles/role_name/meta/main.yml. This broke the installation. Here’s why.

According to http://docs.ansible.com/ansible/playbooks_roles.html, dependencies listed in meta/main.yml are loaded and executed *before* the rest of the role. This is perfect for executing roles that are, in fact, dependencies. Dependencies get taken care of first. In my friend’s case, the upstream contributor didn’t understand that our role is *not* a dependency. When he moved the role invocation to meta/main.yml, he caused it to execute before its own dependencies had been satisfied. The fix was simple: Move our role back to the main playbook.

By the way, here’s what the dependencies look like in meta/main.yml:

---
dependencies:
  - { role: common, some_parameter: 3 }
  - { role: apache, apache_port: 80 }
  - { role: postgres, dbname: blarg, other_parameter: 12 }

The take-away is that using meta dependencies is yet another interesting way Ansible can be used to create clean playbooks that aren’t cluttered with dependencies.