Juju Charms `basic` Layer Hooks

I have had a difficult time trying to figure out how to properly implement a Juju charm using the newer reactive model. The online documentation is incomplete and sometimes incorrect. There are very few examples available.

We developed a charm that deployed a VM into a MaaS server. Once we got it working, we handed it off to QA. Complaint: When they executed juju remove-application, the container for the charm was destroyed, but the VM persisted. They asked me to add support for destroying the VM, too. Sounds reasonable.

If you’ve played with reactive charms, you know that they are built from layers. Each layer provides its own set of capabilities. Most reactive charms are built on the basic layer. I searched a bit and found that there is a set of hooks available in the basic layer:

  • config-changed
  • install
  • leader-elected
  • leader-settings-changed
  • start
  • stop
  • upgrade-charm
  • update-status
  • pre-series-upgrade
  • post-series-upgrade

Each of these hooks is called when certain events or conditions occur with the charm. In the context of the task at hand, the stop hook gets called when the application is being removed, a.k.a. destroyed. The solution, then, is to create a function in reactive/app-name-here.py. The difficult part of the solution is finding out what the code should look like.

After much searching and trial and error, I came across something in the github repository for the reactive package itself. It ended up being a simple, two-line solution in reactive/app-name-here.py:

...
from reactive import hook
...

...
@hook('stop')
def my_destroy_function():
    # Define function here
...

That’s it. All that searching and failing. All one needs to do is import reactive.hook, then decorate the appropriate functions with the name of the hooks they react to. Yeesh. I wish the Juju Charms documentation included practical examples.

Advertisements

Declare Subordinate Charm in Juju Bundle

I’ve been working with Juju Charms a bit. I’ve found that the documentation is quite lacking in specific examples of simple things. For example, I wanted to create a `subordinate` charm. I found a bit of documentation here, and a text hint about what to do there, then I talked to an expert who explained it. It’s really simple! Only there are no straightforward examples out there! Here’s my bundle:

machines:
  "1":
    series: bionic
    constraints: "tags=mytag"
services:
  ubuntu:
    charm: cs:bionic/ubuntu
    num_units: 1
    to:
      - 1
  mycharm:
    charm: /home/user/mycharm
    num_units: 0
    options:
      first_option: "first"
      second_option: "second"
relations:
  - - "ubuntu"
    - "mycharm"

The key is to specify the subordinate charm (mycharm) with num_units: 0 and *do not* include the to: directive.

Also a bit unclear in the docs is the way to specify a subordinate charm metadata.yaml. Here’s mine:

name: mycharm
display-name: MyCharm
summary: Charm to do my stuff
maintainer: Me <me@mydomain.net>
description: |
  Charm to do my stuff
tags:
  - ops
subordinate: true
series:
  - bionic
requires:
  container:
    interface: juju-info
    scope: container

Another few facts that are scattered in the documentation:

 

  • A subordinate charm must have at least one requires relation
  • The relation must be of scope container
  • The provides end of the juju-info interface is present by default in all non-subordinate charms. If you try to specify it, you will get an error.

pipdeptree

I was chasing a dependency issue when executing Python code. After much searching and several blind alleys in that search, evidence started to mount that the problem was related to a package dependency not being properly satisfied.

I started using pip show and other commands to piece together the evidence I needed to pinpoint the error. Then I stumbled upon a python tool that does the work for me: pipdeptree.

pip install pipdeptree

Then execute on a package:

[root@jen-master ~]# pipdeptree -p ansible
ansible==2.4.0.0
- cryptography [required: Any, installed: 2.3.1]
- asn1crypto [required: >=0.21.0, installed: 0.24.0]
- cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- enum34 [required: Any, installed: 1.1.6]
- idna [required: >=2.1, installed: 2.7]
- ipaddress [required: Any, installed: 1.0.22]
- six [required: >=1.4.1, installed: 1.11.0]
- jinja2 [required: Any, installed: 2.10]
- MarkupSafe [required: >=0.23, installed: 1.1.0]
- paramiko [required: Any, installed: 2.4.1]
- bcrypt [required: >=3.1.3, installed: 3.1.4]
- cffi [required: >=1.1, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- six [required: >=1.4.1, installed: 1.11.0]
- cryptography [required: >=1.5, installed: 2.3.1]
- asn1crypto [required: >=0.21.0, installed: 0.24.0]
- cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- enum34 [required: Any, installed: 1.1.6]
- idna [required: >=2.1, installed: 2.7]
- ipaddress [required: Any, installed: 1.0.22]
- six [required: >=1.4.1, installed: 1.11.0]
- pyasn1 [required: >=0.1.7, installed: 0.4.4]
- pynacl [required: >=1.0.1, installed: 1.3.0]
- cffi [required: >=1.4.1, installed: 1.11.5]
- pycparser [required: Any, installed: 2.19]
- six [required: Any, installed: 1.11.0]
- PyYAML [required: Any, installed: 3.13]
- setuptools [required: Any, installed: 0.9.8]

As you can see, a recursive list of every dependency along with version requirements. Very helpful!

Unable to open shell error

Modern versions of Ansible are equipped with a quiver of Network CLI modules that can be used to talk to network devices, e.g. routers, that do not support ssh and python, Ansible’s default method of communication and manipulation of remote hosts. These modules have names such as sros_command, junos_command, and openvswitch_db. You can find the complete list here.

Under the covers, Ansible uses the python paramiko package to connect with these various network devices. Unfortunately, paramiko error handling in the Ansible network modules is lacking. Often, we end up with a catch-all error of the following form:

2018-10-27 15:42:48,890 p=17607 u=root |  fatal: [vsc2.mgmt.vnspoc.net -> localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to open shell. Please see:https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell"
}

We have seen this error caused by many things.

Paramiko version

We have found that some of the unable to open shell errors have been caused by using the wrong version of paramiko. In our testing, paramiko versions 2.2.1 and 2.4.1 work well. You can check your version with:

pip list | grep paramiko

And install the proper version with:

pip install paramiko==2.4.1

Socket handling

As a performance enhancement, Ansible tries to keep open the socket used to connect to your network device. By not repeatedly opening and closing the socket they achieve better performance. We have found that some versions of Ansible do not manage this properly. We have tested Ansible version 2.4.0. It has shown itself to be reliable in this regard. We will be upgrading to 2.7 soon. Fingers crossed…

You can check your version of Ansible with:

ansible --version

And install the proper version with:

pip install ansible==2.4.0.0

You can test if this is your problem by manually deleting the socket file and re-trying. The socket files are located here:

/root/.ansible/pc/

Other packages

We are starting to see that the unable to open shell error is coming more frequently due to paramiko dependency packages being of the wrong version. The only way to determine this is to follow the advice in the error message and set ANSIBLE_DEBUG=true in your environment. This will produce extraordinarily verbose output. Most of that output isn’t useful, but when there is a package mismatch it can be the only way to get to the bottom of it. In one recent case, we found this output in the ansible.log file after enabling debugging:

2018-12-13 11:57:41,783 paramiko.transport Unknown exception: cannot import name certificate_transparency
2018-12-13 11:57:41,785 paramiko.transport Traceback (most recent call last):
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 1925, in run
2018-12-13 11:57:41,785 paramiko.transport     self.kex_engine.parse_next(ptype, m)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/kex_gex.py", line 91, in parse_next
2018-12-13 11:57:41,785 paramiko.transport     return self._parse_kexdh_gex_reply(m)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/kex_gex.py", line 263, in _parse_kexdh_gex_reply
2018-12-13 11:57:41,785 paramiko.transport     self.transport._verify_key(host_key, sig)
2018-12-13 11:57:41,785 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 1747, in _verify_key
2018-12-13 11:57:41,785 paramiko.transport     key = self._key_info[self.host_key_type](Message(host_key))
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib/python2.7/site-packages/paramiko/rsakey.py", line 62, in __init__
2018-12-13 11:57:41,786 paramiko.transport     ).public_key(default_backend())
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.hazmat.backends.openssl.backend import backend
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.hazmat.backends.openssl.backend import backend
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 16, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography import utils, x509
2018-12-13 11:57:41,786 paramiko.transport   File "/usr/lib64/python2.7/site-packages/cryptography/x509/__init__.py", line 7, in 
2018-12-13 11:57:41,786 paramiko.transport     from cryptography.x509 import certificate_transparency
2018-12-13 11:57:41,786 paramiko.transport ImportError: cannot import name certificate_transparency
2018-12-13 11:57:41,786 paramiko.transport 
2018-12-13 11:57:41,788 p=9657 u=root |  connecting to host 10.10.62.117 returned an error
2018-12-13 11:57:41,788 p=9657 u=root |  cannot import name certificate_transparency

 

It’s pretty clear that this is a paramiko error. And all that text boils down to this error:

cannot import name certificate_transparency

A quick search of the Internet for that error shows that when this particular error is encountered the python cryptography library is often out of date. The recommended fix is to update the package:

sudo pip install –upgrade cryptography

What is outlined here is a specific fix for a specific occurrence. There is a general rule illustrated by this specific example. That rule is that paramiko errors are not handled well by paramiko and Ansible. When the generic “unable to open shell” appears, it seems the problem could be caused by a missing or out-of-date python package paramiko relies on. In such a case, enable ANSIBLE_DEBUG and parse the ansible.log.

Why not JavaScript *Without* a Framework?

For months now I have been thinking about the design of a web front end for a collection services. There are *SO MANY CHOICES* out there that it’s sometimes difficult to turn off the incoming streams and make a choice. The exercise of thinking about a design often starts with thinking about frameworks. Code frameworks, primarily written for Javascript with a few for Python, do a bunch of work for you. But there’s a price. Frameworks are often large and regularly force you to implement things in one and only one way. Frameworks are not compatible with one another, so a choice locks you into a heavy investment. Should we use NodeJS? React? Flask? The list is seemingly endless.

Yesterday, I stumbled upon a podcast that takes an alternate approach. The guest on the show, Chris Ferdinandi, asserts that the conversation need not start with a framework. He acknowledges the fact that it can be overwhelming to thinking about frameworks. In some cases, it’s better to just use JavaScript directly. The code is smaller. There’s no framework lock-in. And JavaScript has come a long way from the state it was in several years ago that led to the proliferation of frameworks for it. To help people figure out what to do, Ferdinandi created a website, Go Make Things, where we can find small guides to help us learn to do things *without* resorting to frameworks.

The podcast uses quite a bit of jargon. I didn’t understand it all. But I did find the exposure to another way of thinking to be very helpful. I plan to check out Ferdinandi’s website.

Go Make things:

https://gomakethings.com/

Hanselminutes podcast:

https://www.hanselminutes.com/598/maybe-just-use-vanilla-javascript-with-chris-ferdinandi

AnsibleFest 2017, Network Modules in 2.3, and Ansible connection type

I had the opportunity to attend AnsibleFest 2017 in San Francisco. The sessions I attended were of high quality. It was well worth the cost of attendance.

One of the cool things the Ansible folks setup were “Ask an Expert” sessions. Attendees could sign up for a 15-minute appoitment to discuss whatever issues have been troubling you. I signed up for a session with a network-module author to talk about my recent struggles running network modules using Ansible 2.3.

In Ansible 2.3, network modules require connection: local. When I ran my existing playbooks using Ansible 2.3, I saw errors of the form:

invalid connection specified, expected connection=local, got smart

I spent a bunch of time online before AnsibleFest trying to find out why this was happening. I didn’t understand connection: local, and the documentation was vague. The expert at AnsibleFest set me straight. The explanation was so simple that I had a solid “duh!” moment.

connection: local simply means that the module being invoked will be run on the Ansible node. That’s it. The default, connection: remote, pushes the module’s Python code to the inventory host and executes it there. Starting in Ansible version 2.3, network modules enforce connection: local because they operate against inventory hosts that usually don’t have Python installed, e.g. SROS.

I’ve now adopted the following best practice:

  • Configure my inventory to set connection type for localhost to ‘local’
    • [local_host]
      localhost ansible_connection=local
  • Delegate tasks that I want to run on the Ansible host to localhost
    • delegate_to: localhost

 

Ansible map() not available on el6

If you are running Ansible playbooks on an el6 machine and you run across an error like this:

2017-05-24 10:21:47,585 p=8525 u=root |  fatal: [localhost]: FAILED! =>
{"failed": true, "msg": "The conditional check '( myvsds is defined and 
( myvsds | map(attribute='target_server_type') | list | issuperset([\"kvm\"]) 
or myvsds | map(attribute='target_server_type') | list | issuperset([\"heat\"])
) ) or ( myvcins is defined and ( myvcins | map(attribute='target_server_type')
| list | issuperset([\"kvm\"]) or myvcins | map(attribute='target_server_type')
| list | issuperset([\"heat\"]) ) )' failed. The error was: template
error while templating string: no filter named 'map'. String: {% if (
myvsds is defined and ( myvsds | map(attribute='target_server_type') | list |
issuperset([\"kvm\"]) or myvsds | map(attribute='target_server_type') | list |
issuperset([\"heat\"]) ) ) or ( myvcins is defined and ( myvcins |
map(attribute='target_server_type') | list | issuperset([\"kvm\"]) or myvcins |
map(attribute='target_server_type') | list | issuperset([\"heat\"]) ) ) %} True
{% else %} False {% endif %}\n\nThe error appears to have been in 
'/metro-2.1.1/roles/build/tasks/get_paths.yml': line 8, column 7, but may\nbe
elsewhere in the file depending on the exact syntax problem.\n\nThe offending
line appears to be:\n\n  - block: # QCOW2\n    - name: Find name of VSD QCOW2
File\n      ^ here\n"}

Note the text in BOLD. The problem is caused by the fact that Ansible as of version 2.1 depends on the map() filter implementation from the package python-jinja2. map() was introduced into python-jinja2 starting with python-jinja2 version 2.7. The base python-jinja2 version for el6 is 2.2, thus creating the error, above.

This means that Ansible using map() must be running el7 on the Ansible host.