I’ve been playing with Ansible recently in order to push configuration out to the network, and have been interested in validating the state of the network both before and after a change. I came across Nick Russo’s excellent git repository which does just that.
Testing the state of the network when changes are done manually is usually done by an engineer issuing a number of show commands, and validating that the output is as expected. For instance checking there is an expected number of IS-IS peers and checking all BGP sessions are up. In order to perform this in an automated manner we need to collect the output of the various commands we are interested in, parse them into a format that is understandable by a machine, and then finally execute tests against the output to validate that the output is what we expect.
Setup
For my examples I’m using the following:
- Cisco XRV 9000 running IOS XR 7.0.2
- Ansible 2.9.13
- Cisco IOS XR collection from Ansible Galaxy
I’ve got a basic inventory set up for testing that defines my target.
---
ungrouped:
hosts:
RTR1:
ansible_host: 10.2.0.1
ansible_connection: network_cli
ansible_network_os: iosxr
ansible_user: ansible
ansible_password: password
Gather Data
The first step is to collect output from my target that I want to validate. For this I’m using the iosxr_command module which allows me to run one or more commands on a target, and store the output into a variable.
---
- name: Validate the state of the network
hosts: all
tasks:
- name: Gather data from device
cisco.iosxr.iosxr_command:
commands:
- show isis adjacency
register: "CLI_OUTPUT"
This task logs onto the target, and issues a command. It then registers the output into the variable CLI_OUTPUT for use in further tasks.
The output of this command looks like the following:
RP/0/RP0/CPU0:RTR1#sho isis adjacency
Sat Oct 3 22:08:54.587 UTC
IS-IS ZEN Level-2 adjacencies:
System Id Interface SNPA State Hold Changed NSF IPv4 IPv6
BFD BFD
RTR2 BE1 *PtoP* Up 26 1d02h Yes None None
Total adjacency count: 1
Parse Data
The next step is to parse the output into something that can easily be tested. This is done with the help of writing a filter plugin for ansible. Unfortunately the documentation for this feature is pretty lacking, but they are actually pretty straightforward to implement.
Here is the filter plugin I’ve created to parse this output. Most of the work is done by a regular expression. I’m not going to go over how the regex works, as there is loads of documentation on them out there, but I will point towards the following website which is excellent for helping compose them quickly.
#!/usr/bin/python3
import re
class FilterModule(object):
# This function is required to register filters within ansible.
# It is simply a dictionary containing the name of the filter
# as it will be referenced in a playbook, followed by the name
# of the function to be called
@staticmethod
def filters():
filters = {
"iosxr_parse_isis_neighbours": FilterModule.iosxr_parse_isis_neighbours
}
return filters
# This function is the part that does all the heavy lifting. It parses the
# output of the show command using a regular expression, then adds
# it to an array to be returned back to ansible.
# Each function is called by ansible with a single variable, which is the
# text to be passed in for filtering.
@staticmethod
def iosxr_parse_isis_neighbours(text):
# Define a regular expression that matches the various elements of each
# line of output we are interested in.
pattern = r"""
(?P<system>\S+)\s+
(?P<interface>\S+)\s+
(?P<SNPA>\S+)\s+
(?P<state>\S+)\s+
(?P<hold>\d+)\s+
(?P<changed>\S+)\s+
(?P<NSF>\S+)\s+
(?P<bfdv4>\S+)\s+
(?P<bfdv6>\S+)
"""
# Compile the regex string. The VERBOSE flag allows for whitespace
# and comments to be added into the regex for readability which are
# ignored upon compilation.
regex = re.compile(pattern, re.VERBOSE)
# Define dictionary that will contain the data to be passed back to ansible.
isis_neighbours = []
# Loop over each line of the text passed in, and run the regex over it.
for line in text.split("\n"):
match = regex.search(line)
# If there is a match, add the captured variables into the dictionary
# to be returned to ansible.
if match:
# Any additional processing of the data can be done here if required.
gdict = match.groupdict()
isis_neighbours.append(gdict)
# Return the array of adjacencies back to ansible.
return isis_neighbours
To keep things vaguely organised, I’ve placed this script into a directory plugins/filter. We then need to update the ansible configuration in order to load the filter. Add the following to ansible.cfg
[defaults] filter_plugins = plugins/filter
We can now use this new filter in our playbook to parse the command output
---
- name: Validate the state of the network
hosts: all
tasks:
- name: "Run my filter"
set_fact:
ISIS_NEIGHBOURS: "{{ CLI_OUTPUT.stdout[0] | iosxr_parse_isis_neighbours }}"
Validate Output
We are now ready to run some tests against the output. We do this using the assert module.
---
- name: Validate the state of the network
hosts: all
tasks:
- name: "Ensure router has 1 ISIS neighbours"
assert:
# ISIS_NEIGHBOURS is an array with each element being an
# adjacency. Find out the length of the array to find out how
# many adjacencies there are, and compare it to the expected
# number.
that: "ISIS_NEIGHBOURS | length == 1"
msg: |-
Expected neighbour count did not match actual neighbours
seen. Expected: 1 Saw: {{ ISIS_NEIGHBOURS | length }}
If we now run the playbook, we can see the tests passing. Due to the router having a single IS-IS adjacency, the test passes and everything shows as green.
Altering the number of expected IS-IS peers demonstrates a failure.
Defining the number of expected IS-IS neighbours in the task itself is not particularly useful as each router may have a different number of neighbours. Instead, we can define the expected result for each router in the inventory instead.
---
ungrouped:
hosts:
RTR1:
ansible_host: 10.2.0.1
ansible_connection: network_cli
ansible_network_os: iosxr
ansible_user: ansible
ansible_password: password
isis_neighbours: 2
We can then use variable from the inventory in the test to make it dynamically populate the test on a per neighbour basis.
- name: Validate the state of the network
hosts: all
tasks:
- name: "Ensure router has {{ isis_neighbours }} ISIS neighbours"
assert:
# ISIS_NEIGHBOURS is an array with each element being an
# adjacency. Find out the length of the array to find out how
# many adjacencies there are, and compare it to the expected
# number.
that: "ISIS_NEIGHBOURS | length == {{ isis_neighbours}}"
msg: |-
Expected neighbour count did not match actual neighbours
seen. Expected: {{ isis_neighbours }} Saw: {{ ISIS_NEIGHBOURS | length }}
When running the playbook now, the test is populated with the information pulled out of the inventory file for that specific router.
Looping Over Arrays
There may be occasions where we would like to loop over each of the returned elements from a command, and inspect them individually. We can use ansible’s loops to achieve this.
We will add an additional command to collect from the router to show the status of the BGP neighbours.
---
- name: Validate the state of the network
hosts: all
tasks:
- name: Gather data from device
cisco.iosxr.iosxr_command:
commands:
- show isis adjacency
- show bgp ipv4 unicast summary
register: "CLI_OUTPUT"
The output from this command looks like the following.
RP/0/RP0/CPU0:RTR1#show bgp ipv4 unicast summary Sun Oct 4 20:27:39.540 UTC BGP router identifier 10.0.0.3, local AS number 65432 BGP generic scan interval 60 secs Non-stop routing is enabled BGP table state: Active Table ID: 0xe0000000 RD version: 109 BGP main routing table version 109 BGP NSR Initial initsync version 60 (Reached) BGP NSR/ISSU Sync-Group versions 0/0 BGP scan interval 60 secs BGP is operating in STANDALONE mode. Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer Speaker 109 109 109 109 109 0 Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd 10.0.0.1 0 65432 29 42 109 0 0 00:17:59 0 10.0.0.2 0 65432 36 54 109 0 0 00:12:49 0
We need another filter to process the data from this command. The function is basically the same as the previous one, only with a different regular expression.
#!/usr/bin/python3
import re
class FilterModule(object):
# This function is required to register filters within ansible.
# It is simply a dictionary containing the name of the filter
# as it will be referenced in a playbook, followed by the name
# of the function to be called
@staticmethod
def filters():
filters = {
"iosxr_parse_isis_neighbours": FilterModule.iosxr_parse_isis_neighbours,
"iosxr_parse_bgp_neighbours": FilterModule.iosxr_parse_bgp_neighbours
}
return filters
@staticmethod
def iosxr_parse_bgp_neighbours(text):
pattern = r"""
(?P<peer>\d+\.\d+\.\d+\.\d+)\s+
(?P<spk>\d+)\s+
(?P<remoteas>\d+)\s+
(?P<msgrcvd>\d+)\s+
(?P<msgsnt>\d+)\s+
(?P<tblver>\d+)\s+
(?P<inq>\d+)\s+
(?P<outq>\d+)\s+
(?P<uptime>\S+)\s+
(?P<state>\S+)
"""
regex = re.compile(pattern, re.VERBOSE)
bgp_neighbours = []
for line in text.split("\n"):
match = regex.search(line)
if match:
gdict = match.groupdict()
bgp_neighbours.append(gdict)
return bgp_neighbours
We can now use the filter to parse the output, and then
---
- name: Validate the state of the network
hosts: all
- name: "Run my filter"
set_fact:
ISIS_NEIGHBOURS: "{{ CLI_OUTPUT.stdout[0] | iosxr_parse_isis_neighbours }}"
BGP_IPV4_NEIGHBOURS: "{{ CLI_OUTPUT.stdout[1] | iosxr_parse_bgp_neighbours }}"
- name: "Ensure that all IPv4 BGP peers are established"
assert:
# Ensure that the state is not active or idle. If it is established
# the number of routes received.
that: item.state | lower != 'active' and item.state | lower != 'idle'
msg: |-
Neighbour {{ item.peer }} is not established!
# This is the magic that causes the assert to run for each element
# in the array.
loop: "{{ BGP_IPV4_NEIGHBOURS }}"
loop_control:
# Create a label to output for each element in the array. Without
# this the full object gets printed out.
label: "Peer: {{ item.peer }}, AS Number: {{ item.remoteas }}"
This gives the desired output, but it’s a bit verbose.
We can reduce this by enabling the quiet attribute.
---
- name: Validate the state of the network
hosts: all
- name: "Run my filter"
set_fact:
ISIS_NEIGHBOURS: "{{ CLI_OUTPUT.stdout[0] | iosxr_parse_isis_neighbours }}"
BGP_IPV4_NEIGHBOURS: "{{ CLI_OUTPUT.stdout[1] | iosxr_parse_bgp_neighbours }}"
- name: "Ensure that all IPv4 BGP peers are established"
assert:
# Limit the output of each assert.
quiet: True
# Ensure that the state is not active or idle. If it is established
# the number of routes received.
that: item.state | lower != 'active' and item.state | lower != 'idle'
msg: |-
Neighbour {{ item.peer }} is not established!
# This is the magic that causes the assert to run for each element
# in the array.
loop: "{{ BGP_IPV4_NEIGHBOURS }}"
loop_control:
# Create a label to output for each element in the array. Without
# this the full object gets printed out.
label: "Peer: {{ item.peer }}, AS Number: {{ item.remoteas }}"
Running it now gives a much more compact output.
If we run the playbook now with one of the BGP sessions down, we get a failure as expected. Unfortunately I’ve not discovered a way to reduce the output in a failure, but at least it’s contained to one line.
Conclusion
I’ve found it’s quite quick and easy to create playbooks to validate the state of the network. These can be ran before and after a deployment, and the output compared to ensure that nothing unexpected has happened.
It might be useful to combine this with TextFSM to parse the command output instead of using a plain regular expression.