Use Chrome Tab Groups to Reduce the Size of the Haystack

Posted on by Brian Castelli

Don’t you hate it when your Chrome browser has 1,000 open tabs and you can’t find / remember which one was what you were just looking at a few minutes ago? It’s easier to find a needle in a haystack!

One way to make tabs easier to find is to organize them using Chrome Tab Groups.

Chrome allows you to create these tab groups and move your tabs in and out of them. Here’s an example of some tab groups in action:

The colored tabs with AWS, Jenkins, Confluence, Github, and Jira were created by me, then I moved the appropriate browser tabs into each group. This has made finding tabs so much easier!

To get started, right click on any tab and choose Add Tab to Group → New Group:

As you can see from the image, you can also use this method to move a tab to an existing group.

And, of course, you can drag and drop tabs into and out of existing tab groups.

Whew!

Fix AWS Session Manager Can’t Connect Permissions

Posted on by Brian Castelli

One of our users wanted to be able to access the shell prompt on a specialized AMI instance that he had spun up. He had ssh access via ssh key, but using EC2 → Instances → Specific Instance → Connect to instance was not working for either EC2 Instance Connect or Session Manager.

I was able to diagnose this particular failure as a simple permissions issue on his EC2’s instance role. Lucky for me, EC2 Instance Connect worked with my admin user privs. But I could have used ssh with his ssh key.

Steps:

  1. Verify that the SSM Agent is running on the EC2. In this case, the user had created an EC2 instance running Ubuntu 20.4, requiring the use of snap. Other Linux versions may require a different command.
    • sudo snap list amazon-ssm-agent
      • Name Version Rev Tracking Publisher Notes
        amazon-ssm-agent 3.2.419.0 6783 latest/stable/… aws✓ classic
    • The output in my case indicated that the agent was installed. I could move on to the next step.
    • If your output indicates that the agent isn’t installed, you’ll need to install it.
  2. Cat the SSM agent log
    • sudo more /var/log/amazon/ssm/amazon-ssm-agent.log
    • The output in my case included:
      • User: arn:aws:sts::123456789:assumed-role/ecsInstanceRole/i-123402283adf2501 192 is not authorized to perform: ssm:UpdateInstanceInformation on resource...
  3. Open IAM → Roles and find the assumed-role name
  4. Add the AmazonSSMManagedInstanceCore policy which includes the required "ssm:UpdateInstanceInformation" permission.
  5. Try Session Manager again.

Adding Users to Manage Kubernetes Cluster on AWS EKS

Posted on by Brian Castelli

AWS EKS support for kubernetes has some quirks. The most-troublesome one is the fact that only the entity that creates a cluster can manage it. For example, if Amir creates a cluster using his AWS IAM credentials and promptly resigns, no one else in your company can manage that cluster. Even AWS support will not be able to delete it – or so they told me.

The simplest method I have found for making a cluster manageable by others is to edit the aws-auth config map for the cluster. You can add individual users and roles to the system:masters group for the cluster. (I am partial to adding roles and then managing role membership as a way to control who can access the cluster…)

Pulling together advice from a few web sites, this is the summary of what I am doing:

Prerequisites

  • AWS CLI
  • Kubectl
  • Eksctl
  • Default editor configured on your shell

Steps

  • Ensure that you are using the proper identity and AWS credentials
  1. Use aws sts get-caller-identity to ensure that you are using the proper credentials
  2. You can change credentials using aws configure
  3. Use eksctl utils write-kubeconfig --cluster <cluster-name> to configure your kubeconfig for the cluster you want to add users to.
  4. Use kubectl edit configmap aws-auth --namespace kube-system to modify the aws-auth ConfigMap for the cluster. Your default editor will open. On a brand-new cluster, the mapUsers section of the file will be empty. Add users following the YAML format shown in the mapUsers section, below, as a guide. Or add a role to the mapRoles section.

    You should not edit any other part of the file – in fact, we are warned that messing up this file can lead to everyone being locked out of the cluster. Proceed with caution. the values shown, below, are for example purposes only. DO NOT COPY THEM.

    apiVersion: v1
    data:
    mapRoles: |
    - groups:
    - system:bootstrappers
    - system:nodes
    rolearn: arn:aws:iam::nnnnnnnnnnnn:role/cluster-name-nodegroup-ng-NodeInstanceRole-BL7FPEHQ
    username: system:node:{{EC2PrivateDNSName}}
    mapUsers: |
    - userarn: arn:aws:iam::nnnnnnnn:user/user.name
    username: user.name
    groups:
    - system:masters
    - userarn: arn:aws:iam::nnnnnnnn:user/user.name2
    username: user.name2
    groups:
    - system:masters
    kind: ConfigMap
    metadata:
    creationTimestamp: "2022-11-24T22:07:34Z"
    name: aws-auth
    namespace: kube-system
    resourceVersion: "1577569"
    uid: 123456-e129-4fe0-9ecb-246425fba343
     
  5. Save the file. kubectl will take care of updating the cluster. It may take a minute or two, but eventually the users /roles you added will be able to manage the cluster.

Revert tar.gz Bucket Objects in AWS S3

Posted on by Brian Castelli

Good news: One can enable versioning on Amazon AWS S3 buckets!

Not so good news: AWS Console offers no direct, easy way to revert an object to a previous version.

Work around: Download the old version and upload it back.

Sounds simple. Unless, as in my case, the ‘object’ was actually a tar.gz archive.

Here’s what happened: Using the AWS Console and the aws s3 cli, the downloaded file lost the gz extension. my_file.tar.gz ended up on disk as my_file.tar. Hmmm.

Workaround: Use the aws s3api command to download.

My process:

  1. Find the version id of the object / file you want to restore. You can do this via the AWS Console or using the aws s3api command. The following command will give you the version ids of every object in the bucket:
    aws s3api list-object-versions --bucket my-bucket
  2. Copy the version of the file you want from S3 to your computer:
    aws s3api get-object --bucket my-bucket --key path-to-file/my_file.tar.gz --version-id 12345ABFPhsFitffwuu9Q6iHOUtx0hID my_file.tar.gz
  3. Copy the file back to the bucket (aws s3 cp works fine here…)
    aws s3 cp ./my_file.tar.gz s3://my-bucket/path-to-file/

Quiet

Posted on by Brian Castelli

Quiet, a book by Susan Cain, was quite an eye opener for me. Cain writes variously about the creative genius often present in the introverts among us, the strengths and weaknesses of both introverts and extroverts, and the bias in western culture towards extroverts. We perceive those who exhibit extroversion to be smarter, better, and more desirable. Yet statistics show that introverts, when not forced into situations better suited for extroverts, are quite creative and hard working. They foster deep friendships with a few. They are known for deep, careful thinking about difficult problems. They are known to persevere when their extroverted cousins would have quit and moved on. 

Among the most-surprising of the research results in Cain’s book is the discussion of stimulation. Referencing the work of research psychologist Hans Eysenck, Cain points out that the difference between an introvert and extrovert is primarily the amount of stimulation they are comfortable with. A snippet from the book:

For several decades, beginning in the late 1960s, an influential research psychologist named Hans Eysenck hypothesized that human beings seek “just right” levels of stimulation—not too much and not too little. Stimulation is the amount of input we have coming in from the outside world. It can take any number of forms, from noise to social life to flashing lights. Eysenck believed that extroverts prefer more stimulation than introverts do, and that this explained many of their differences: introverts enjoy shutting the doors to their offices and plunging into their work, because for them this sort of quiet intellectual activity is optimally stimulating, while extroverts function best when engaged in higher-wattage activities like organizing team-building workshops or chairing meetings.

 The implications of stimulus are huge. This is why introverts thrive when they have their own office while extroverts thrive in open-office environments. We can’t have a one-size-fits-all office environment. People must have both places to collaborate and places to do deep thinking without distractions.

If you’re a manager, there’s another implication: give your team as much uninterrupted time as possible. A Slack message, a meeting in the middle of the afternoon, and an unscheduled visit to their cube all represent stimulation.

Leadership and Management

Posted on by Brian Castelli

Covey succinctly captures a truth: Leadership and management are two different things. As with so many other things, I believe that people are most often not one thing or the other but a blend of the two. I don’t know anyone who is a pure leader, doing no management. And vice-versa.

I am a manager by title and by practice. It’s my job to help my team put “first things” first as often and as consistently as they can. In that work, however, I must also lead. I determine, for example, that having Sprint Planning every second Wednesday is a “first thing” the team must put on the top of their priority list. When Sprint Retrospectives aren’t being effective, I make it a “first thing” for the team to iterate on changes that will improve Retrospective effectiveness.

The best managers I know are also leaders in the context of their teams.

Strengths into Excellence

Posted on by Brian Castelli

In their practical and fascinating book Nine Lies About Work, Marcus Buckingham and Ashley Goodall make a strong case for achieving excellent performance by building team member’s strengths, not fixing their weaknesses. They sight examples from the sporting world (Messi) and politics (Churchill) of people who turned their strengths into excellence.

What does that mean for team leaders?

If we spend our time trying to turn weaknesses into strengths, we will get good team members. Good but not great. To achieve great, focus on what they already do well, help them hone and refine it, steer work their way that they can excel at, and everybody is happy.

Chasing Ansible 2.9 Paramiko Error on Exit

Posted on by Brian Castelli

needle-in-a-haystack-1752846_1920

As a heavy user of Ansible, I am always concerned about updating to the latest version in order to have the most-recent security features and patches. With a user base of over 500, pushing updates to everyone almost always causes someone to trip. We recently updated to Ansible 2.9.0. A few users started seeing unusual python exceptions related to one of the packages Ansible depends on: Paramiko. For example:

ansible --version
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 120, in _join_lingering_threads
    for thr in _active_threads:
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 120, in _join_lingering_threads
    for thr in _active_threads:
TypeError: 'NoneType' object is not iterable

I used a *very* unscientific method to get this user up and running. I have a testbed in my lab that is working great with Ansible 2.9. I used the command `pip show` to walk through the dependencies of Ansible. Once I compiled a list of the versions of the dependencies 2 levels down, I compared them with the user’s machine. There were a few differences. When we used pip to manually modify the installed versions on this user’s machine, everything started working properly.

I offer the following list for what worked for us on CentOS 7 and Ansible 2.9. Your mileage may vary:

ansible==2.9
ipaddr==2.2.0
netaddr==0.7.19
netmiko==2.3.0
paramiko==2.6.0
cryptography==2.8
pynacl==1.3.0
bcrypt==3.1.7
cffi==1.13.1
six==1.9.0
enum34==1.1.6
requests==2.23.0

The main take-away from this post is *not* the needle-in-a-haystack magic combination shown, above. The main point is that package management in Python can be tricky. The version of pip you are using, the version of Python you have installed, and a dozen other factors can come into play. `pip show` is one tool you can use to dissect the problem to do some trial and error – especially when the Internet seems to be silent about the needle you are seeking.

Juju Charms `basic` Layer Hooks

Posted on by Brian Castelli

I have had a difficult time trying to figure out how to properly implement a Juju charm using the newer reactive model. The online documentation is incomplete and sometimes incorrect. There are very few examples available.

We developed a charm that deployed a VM into a MaaS server. Once we got it working, we handed it off to QA. Complaint: When they executed juju remove-application, the container for the charm was destroyed, but the VM persisted. They asked me to add support for destroying the VM, too. Sounds reasonable.

If you’ve played with reactive charms, you know that they are built from layers. Each layer provides its own set of capabilities. Most reactive charms are built on the basic layer. I searched a bit and found that there is a set of hooks available in the basic layer:

  • config-changed
  • install
  • leader-elected
  • leader-settings-changed
  • start
  • stop
  • upgrade-charm
  • update-status
  • pre-series-upgrade
  • post-series-upgrade

Each of these hooks is called when certain events or conditions occur with the charm. In the context of the task at hand, the stop hook gets called when the application is being removed, a.k.a. destroyed. The solution, then, is to create a function in reactive/app-name-here.py. The difficult part of the solution is finding out what the code should look like.

After much searching and trial and error, I came across something in the github repository for the reactive package itself. It ended up being a simple, two-line solution in reactive/app-name-here.py:

...
from reactive import hook
...

...
@hook('stop')
def my_destroy_function():
    # Define function here
...

That’s it. All that searching and failing. All one needs to do is import reactive.hook, then decorate the appropriate functions with the name of the hooks they react to. Yeesh. I wish the Juju Charms documentation included practical examples.

Declare Subordinate Charm in Juju Bundle

Posted on by Brian Castelli

I’ve been working with Juju Charms a bit. I’ve found that the documentation is quite lacking in specific examples of simple things. For example, I wanted to create a `subordinate` charm. I found a bit of documentation here, and a text hint about what to do there, then I talked to an expert who explained it. It’s really simple! Only there are no straightforward examples out there! Here’s my bundle:

machines:
  "1":
    series: bionic
    constraints: "tags=mytag"
services:
  ubuntu:
    charm: cs:bionic/ubuntu
    num_units: 1
    to:
      - 1
  mycharm:
    charm: /home/user/mycharm
    num_units: 0
    options:
      first_option: "first"
      second_option: "second"
relations:
  - - "ubuntu"
    - "mycharm"

The key is to specify the subordinate charm (mycharm) with num_units: 0 and *do not* include the to: directive.

Also a bit unclear in the docs is the way to specify a subordinate charm metadata.yaml. Here’s mine:

name: mycharm
display-name: MyCharm
summary: Charm to do my stuff
maintainer: Me <me@mydomain.net>
description: |
  Charm to do my stuff
tags:
  - ops
subordinate: true
series:
  - bionic
requires:
  container:
    interface: juju-info
    scope: container

Another few facts that are scattered in the documentation:

 

  • A subordinate charm must have at least one requires relation
  • The relation must be of scope container
  • The provides end of the juju-info interface is present by default in all non-subordinate charms. If you try to specify it, you will get an error.