== '''Network Device Configuration Management - Oxidized ''' ==

In this lab, you will install and configure Oxidized and integrate it with LibreNMS.''' When you are changing users, always start from root.''' You may use '''exit''' to logout current user. When you are coping the commands, copy and execute line by line. Otherwise, you may get errors and worst case, have to start from the beginning.

'''Requirements:'''

1. Completed network setup of [https://ws.learn.ac.lk/wiki/Cnbp2022/Agenda/LabSetup] (Download '''WKFiles''' files from this link and replace WKFiles with it in your setup [https://docs.learn.ac.lk/index.php/s/S2n5VymHTzQiZzS])

•  Configured PFsense VM. '''If PFsense in your setup is incomplete''', replace it with this VM and adjust network configurations if required. We recommend using the wired network as much as possible. [https://docs.learn.ac.lk/index.php/s/xNKnIMShvRdUFWV]

2. Ubuntu 20.04 Server with LibreNMS installed (Use this provided VM to replace '''Monitoring_Server''' in your setup [https://docs.learn.ac.lk/index.php/s/yWmI7pZCIV8lf9o])[[BR]]

3. Gmail account with 2-Step Verification ''On''.


'''Oxidized Installation'''

Login/SSH to LibreNMS VM (10.1.5.10) ''(username:user & password:1234)'' from  WinXP_INSIDE. Since most of the tasks in this lab require you to be "root", the first thing you should do is to connect to your server and start a root shell.

{{{
sudo su
apt update
apt-get install ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libssh2-1-dev libicu-dev zlib1g-dev g++
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.lo.disable_ipv6=1
gem install oxidized
gem install oxidized-script oxidized-web 
}}}

You may run '''oxidized''' command (below last command) 2 to 3 times to generate all configurations. It is recommended '''not to''' run oxidized as root.

{{{
useradd -s /bin/bash -m oxidized
su - oxidized
oxidized
}}}


'''LibreNMS Integration'''

''Create Token as follows''

Login to LibreNMS ''(username & password:admin)'' web GUI (10.1.5.10) using WinXP_INSIDE.
Go to Settings -> API Settings[[BR]]

[[Image(librenms api settings.png)]][[BR]]

Click on 'Create API access token'.[[BR]]

[[Image(librenms create token.png)]][[BR]]

Click 'Create API Token'.

Login to CLI as libreNMS user and continue.

{{{
sudo su - librenms
lnms config:set oxidized.enabled true
lnms config:set oxidized.url http://127.0.0.1:8888
lnms config:set oxidized.features.versioning true
lnms config:set oxidized.group_support true
lnms config:set oxidized.default_group f
}}}


'''Oxidized Configuration'''

Open oxidized main configuration file.

{{{
su - oxidized
nano .config/oxidized/config
}}}

Change the configuration file by adding / updating the following information. Replace '''oxidizedfor''' (gmail) and '''09db454fb0ec6927c777e214c2f82d60''' (LibreNMS token  value) with your corresponding values. ''Interval'' value (in seconds) is the ''auto refresh interval'' (you can change that one).
 
{{{
---
username: username
password: password
model: ios
resolve_dns: false
interval: 300
use_syslog: false
debug: false
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: 127.0.0.1:8888
next_adds_job: false
vars:
  enable: '123'
groups: {}
models: {}
pid: "/home/oxidized/.config/oxidized/pid"
crash:
  directory: "/home/oxidized/.config/oxidized/crashes"
  hostnames: false
stats:
  history_size: 10
input:
  default: telnet, ssh
  debug: false
  ssh:
    secure: false
  ftp:
    passive: true
  utf8_encoded: true
output:
  default: git
  git:
    user: Oxidized
    email: oxidizedfor@gmail.com
    repo: "~/.config/oxidized/oxidized.git"
source:
  default: http
  debug: true
  http:
    url: http://127.0.0.1/api/v0/oxidized
    map:
      name: hostname
      model: os
      group: group
    headers:
      X-Auth-Token: '09db454fb0ec6927c777e214c2f82d60'
groups:
  f:
    username: 'cisco'
    password: '123'
hooks:
  email_output:
    type: exec
    events: [post_store]
    cmd: '/home/oxidized/.config/oxidized/oxidized-report-git-commits -s "Oxidized updates for ${OX_NODE_NAME}" -r oxidizedfor@gmail.com'
    async: true
    timeout: 120
}}}

Create oxidized-report-git-commits file.

{{{
nano .config/oxidized/oxidized-report-git-commits
}}}

Copy the following content to the file and save it.

{{{
#!/bin/sh
trap '/bin/rm -f "$tmpfile"' EXIT

tmpfile=$(mktemp) || exit 1

subject="Oxidized updates for ${OX_NODE_NAME}"
scriptname=`basename $0`

usage()
{
    echo "Usage: ${scriptname} [-f] [ -s email_subject ] [ -r email_recipient ]"
    exit 1
}

email_on_gitfail=1
while getopts "fs:r:" opt; do
    case $opt in
        s)
            subject=$OPTARG
            ;;
        r)
            recipient=$OPTARG
            ;;
        f)
            email_on_gitfail=0
            ;;
        *)
            usage
            ;;
    esac
done


if [ "${OX_EVENT}" = "node_fail" ]; then
    echo "${scriptname}: ${OX_NODE_NAME}": 'Job failed'
    exit 64
fi

if [ -z "${OX_REPO_COMMITREF}" ]; then
        echo "${scriptname}: "'$OX_REPO_COMMITREF not set'
        exit 64
fi

if [ -z "${OX_REPO_NAME}" ]; then
        echo "${scriptname}: "'$OX_REPO_NAME not set'
        exit 64
fi

cat > ${tmpfile} <<EOF
Node name: ${OX_NODE_NAME}
Group name: ${OX_NODE_GROUP}
Job status: ${OX_JOB_STATUS}
Job time: ${OX_JOB_TIME}
Git repo: ${OX_REPO_NAME}
Git commit ID: ${OX_REPO_COMMITREF}

EOF

# test if commit exists
git --bare --git-dir="${OX_REPO_NAME}" rev-parse --quiet --verify "${OX_REPO_COMMITREF}" > /dev/null 2>&1
gitret=$?

if [ ${gitret} -eq 0 ]; then
    git --bare --git-dir="${OX_REPO_NAME}" show --pretty='' --no-color "${OX_REPO_COMMITREF}" >> ${tmpfile} 2>&1
else
    echo "${scriptname}: commit ${OX_REPO_COMMITREF} does not exist" >> ${tmpfile}
fi

if [ ! -z "${recipient}" -a \( ${gitret} -eq 0 -o ${email_on_gitfail} -eq 1 \) ]; then
    cat ${tmpfile} | mail -s "${subject}" "${recipient}"
else
    cat ${tmpfile}
fi
}}}

Give execution privileges. 

{{{
chmod +x .config/oxidized/oxidized-report-git-commits
}}}

'''Setup Oxidized Service'''

Run the following commands as root user.

{{{
cp /var/lib/gems/2.7.0/gems/oxidized-0.28.0/extra/oxidized.service /etc/systemd/system
mkdir /run/oxidized
chown oxidized:oxidized /run/oxidized
systemctl enable oxidized.service
}}}

Start oxidised service 

{{{
systemctl start oxidized
}}}

View status of oxidised service

{{{
systemctl status oxidized
}}}

Login to LibreNMS GUI by browsing 10.1.5.10 in the web using WinXP_INSIDE. 

Oxidized can be accessed as in the following.[[BR]]

[[Image(oxidised librenms.png, 600px)]][[BR]]

A device configuration can be accessed by clicking the icon (see the below red arrow). [[BR]]

[[Image(librenms oxidized.png, 1200px)]][[BR]]

The resulting screen will be as below or similar.

[[Image(device config.png, 1200px)]][[BR]]

You can view ''Show version'' and ''Show diff'' buttons '''if there are configuration changes/ changing history, after Oxidized has connected with the devices.''' Try to get that screen by changing a configuration and '''saving''' that of a connected device. All device passwords are ''123''. You may refer this [https://ws.learn.ac.lk/wiki/Cnbp2022/Agenda/NetworkSetup]

The default output will be ''Show version''.

[[Image(history.png, 1200px)]][[BR]]

Click '''Show diff'''. Below '''Hide line numbers''' a minus (-) mark in front of the command says that particular command is reverted ('''red'''). And a plus (+) mark in front of the command says the reverted command is replaced by it ('''green''').

[[Image(diff.png, 1300px)]][[BR]]

'''Postfix Installation'''

Postfix will be installed to have an email notification of configuration changes. Run the following commands as root user.

{{{
apt install postfix
}}}

Select ''Internet Site'' (default option) to enable Postfix to sent and receive mails and press ''Enter'' to proceed.

[[Image(internet site.png, 700px)]][[BR]]

Put '''oxidized.learn.com''' for ''System mail name'' and proceed. 

[[Image(mail name.png, 1700px)]][[BR]]

'''Gmail Configuration'''

Go to [https://myaccount.google.com/] and sign into your Gmail account.

Select Security.

Under "Signing in to Google," select App Passwords.[[BR]] 

Generate app password by selecting custom option as follows and note down the password. [[BR]]

[[Image(select app.png, 600px)]][[BR]]

'''Postfix Configuration'''

Here Postfix will be configured to send email to your Gmail account. First make sure that you are using ''root'' account. 

{{{
vim /etc/postfix/main.cf
}}}

Go through file carefully and add / update the necessary lines as below. Don't fully remove any default line. Some default lines are also depicted in the below configuration to aid positioning lines in the correct place. 

{{{
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_security_level=may

smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level=may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = oxidized
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname, oxidized.learn.com, oxidized, localhost.localdomain, localhost
relayhost = [smtp.gmail.com]:587
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
}}}

{{{
vim /etc/postfix/sasl_passwd
}}}

Add the following content to /etc/postfix/sasl_passwd while changing '''oxidizedfor''' and '''gecqeehdvtrrsed5''' (app password). 
{{{
[smtp.gmail.com]:587 oxidizedfor@gmail.com:gecqeehdvtrrsed5
}}}

Close the above file and continue.
{{{
chown root:root /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
systemctl restart postfix
apt install mailutils
}}}

Check email sending. Replace '''oxidizedfor''' accordingly. Check the gmail inbox (also the spam folder). You may have to refresh your Gmail inbox several times. 
{{{
echo "Test Postfix Gmail SMTP Relay" | mail -s "Postfix Gmail SMTP Relay" oxidizedfor@gmail.com
}}}

An email will be sent to the Gmail account after a saved configuration change, when Oxidized get refreshed.

Such email content will be in a similar format as follows. There the minus (-) mark in front of the command says that particular command is reverted. And plus (+) mark in front of the command says the reverted command is replaced by it. 

[[Image(notification.png)]][[BR]]

'''After completing the lab''', you should email two screenshots as one from a device configuration /''config'' page of Oxidized in LibreNMS and, the other from ''configuration change notification email'', to {{{deepthi@learn.ac.lk}}}. Please mention your 'workshop registration details' in the email.