= SNMP and LibreNMS = In the first part of this Hands-On you will install and learn some basic SNMP commands. Then you will Install LibreNMS and Configure the Network Management System == SNMP == === Installing client (manager) tools === Start by installing the net-snmp tools: {{{ $ sudo apt-get install snmp $ sudo apt-get install snmp-mibs-downloader }}} The second of the two commands downloads the standard IETF and IANA SNMP MIBs which are not included by default. Now, edit the file /etc/snmp/snmp.conf: {{{ $ sudo vi /etc/snmp/snmp.conf }}} Change this line: {{{ mibs : }}} ... so that it looks like: {{{ # mibs : }}} (You are “commenting out” the empty mibs statement, which was telling the snmp* tools not to automatically load the mibs in the /usr/share/mibs/ directory) Now, in your home directory make a .snmp directory with file snmp.conf inside it, make it readable only be you, and add the credentials to it: {{{ $ cd $ mkdir .snmp $ chmod 700 .snmp/ $ vi .snmp/snmp.conf }}} Put the following contents in the file: {{{ defVersion v2c defCommunity NetManage }}} Creating this configuration file means you won't have to enter your snmp community everytime you use one of the SNMP utilities. === Configure SNMP on Your Router (Already done) === '''Note''': If you are using a cisco router this is the way to enable snmp on the device. In this lab this part is already done. Connect to your router: {{{ rtrN> enable Password: <> rtrN# configure terminal (conf t) }}} Now we need to add an Access Control List rule for SNMP access, turn on SNMP, assign a read-only SNMP community string as well as a SNMPv3 group and user and tell the router to maintain SNMP information across reboots. To do this we do: {{{ rtrN(config)# access-list 60 permit 192.248.6.0 0.0.0.255 (Allow from workshop IP subnet) rtrN(config)# snmp-server community NetManage ro 60 }}} Now let's exit and save this new configuration to the routers permanent config. {{{ rtrN(config)# exit rtrN# write memory (wr mem) rtrN# exit (until you return to your pc) }}} === Testing SNMP === To check that your SNMP installation works, run the snmpstatus command on each of the following devices {{{ $ snmpstatus 192.248.1.1 }}} Where 192.248.1.1 is the IP of the LEARN Router What happens if you try using the wrong community string (i.e. change !NetManage to something else) using the options "'''-c !NetWrong'''”? === SNMP Walk and OIDs === Now, you are going to use the snmpwalk command, part of the SNMP toolkit, to list the tables associated with the OIDs listed below, on each piece of equipment you tried above: {{{ .1.3.6.1.2.1.2.2.1.2 .1.3.6.1.2.1.31.1.1.1.18 .1.3.6.1.2.1.25.2.3.1 }}} You will try this with two forms of the snmpwalk command: {{{ $ snmpwalk }}} and {{{ $ snmpwalk -On }}} ... where OID is one of the OIDs listed above: .1.3.6... ...where IP_ADDRESS can be your group's router... '''Note''': the -On option turns on numerical output, i.e.: no translation of the OID <-> MIB object takes place. For these OIDs: a) Do all the devices answer ? b) Do you notice anything important about the OID on the output ? === Configuration of snmpd on your www Server === For this exercise your group needs to verify that the snmpd service is running and responding to queries for all machines in your group. First enable snmpd on your machine, then test if your machine is responding, then check each machine of your other group members. Install the SNMP agent (daemon) {{{ $ sudo apt-get install snmpd $ sudo apt-get install libsnmp-dev }}} ==== Configuration ==== We will make a backup of the distributed config, and then we will create our own: {{{ $ cd /etc/snmp $ sudo mv snmpd.conf snmpd.conf.dist $ sudo vi snmpd.conf }}} Then, copy/paste the following: {{{ # Listen for connections on all interfaces (both IPv4 *and* IPv6) agentAddress udp:161,udp6:[::1]:161 # For SNMPv2: Configure Read-Only community and restrict who can connect rocommunity NetManage 192.248.0.0/16 rocommunity NetManage 127.0.0.1 # Information about this host sysLocation LEARN Workshop sysContact admin@'your domain'.ws.learn.ac.lk # Which OSI layers are active in this host # (Application + End-to-End layers) sysServices 72 # Include proprietary dskTable MIB (in addition to hrStorageTable) includeAllDisks 10% }}} Now save and exit from the editor. Now we will add the same SNMPv3 user to your PC. We need to stop snmpd before adding the user, and restart it to read the above changes as well as the new user: {{{ $ sudo service snmpd restart }}} Check that snmpd is working: {{{ $ snmpstatus localhost }}} What do you observe ? === Test your neighbors === Check now that you can run snmpstatus against your other group members servers: {{{ $ snmpstatus www.'your neighbors domain'.ws.learn.ac.lk }}} == LibreNMS == === Installing LibreNMS === These instructions assume you are the '''root''' user. If you are not, prepend ''sudo'' to the shell commands (the ones that aren't at ''mysql>'' prompts) or temporarily invoke root privileges. ==== Create database ==== '''NOTE''': These instructions are based on the official LibreNMS installation notes and have been tested on a fresh install of Ubuntu 16.0. We will assume that the database is running on the same machine as your network management server (this is the most common initial deployment scenario). First install mysql and configure: {{{ # apt-get update (Already done) # apt-get install mysql-server mysql-client }}} You will be asked to enter a password for the MySQL root user. Be absolutely sure that you remember what you choose here. You will use this later. {{{ # mysql -u root -p }}} Input the MySQL root password (the one you chose in the previous step) to enter the MySQL command-line interface where you will get a mysql> prompt. Create the database: {{{ CREATE DATABASE librenms; GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost' IDENTIFIED BY ''; FLUSH PRIVILEGES; exit }}} '''Note''' : Here we are using (Make sure your password match with the policies you add in last lab session eg. Linux@2016) as the password for LibreNMS to access MySQL. Please replace with, Please do not forget the password as you will need it in the future Then edit '''/etc/mysql/mysql.conf.d/mysqld.cnf''' to avoid mysql to work in strict mode {{{ vi /etc/mysql/mysql.conf.d/mysqld.cnf }}} Within the '''[mysqld]''' section please add: {{{ innodb_file_per_table=1 sql-mode="" }}} Restart Mysql service {{{ service mysql restart }}} ==== Install LibreNMS ==== The NMS is the host is where the web server and SNMP poller run. Install the required software: {{{ apt-get install libapache2-mod-php7.0 php7.0-cli php7.0-mysql php7.0-gd php7.0-snmp php-pear php7.0-curl snmp graphviz php7.0-mcrypt php7.0-json apache2 fping imagemagick whois mtr-tiny nmap python-mysqldb snmpd php-net-ipv4 php-net-ipv6 rrdtool git }}} The packages listed above are an all-inclusive list of packages that were necessary on a clean install of Ubuntu 16.0 ==== snmp ==== You need to configure snmpd appropriately if you have not already done so. We will do a minimal snmp configuration on our server -- '''please DON'T''' do this if you've already configured SNMP earlier! {{{ # mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.orig # vi /etc/snmp/snmpd.conf }}} and, add the following line to the empty file: {{{ rocommunity NetManage 127.0.0.1 }}} And, now restart the snmp service so that the changes become active. {{{ # service snmpd restart }}} You can verify that snmp now responds to you locally by typing: {{{ # snmpstatus -v2c -c NetManage 127.0.0.1 sysStatus }}} ==== php ==== In both '''/etc/php/7.0/apache2/php.ini''' and '''/etc/php/7.0/cli/php.ini''', ensure '''date.timezone''' is set to your preferred time zone. See or files under ''/usr/share/zoneinfo'' for a list of supported timezones. For this workshop we are all going to use the same timezone. In the two archives noted above find the line that reads: {{{ ;date.timezone = }}} and change it to: {{{ date.timezone = Asia/Colombo }}} Save and exit from the files. ==== Adding the LibreNMS user ==== We need to create a LibreNMS system user, librenms {{{ # useradd librenms -d /opt/librenms -M -r # usermod -a -G librenms www-data }}} ==== Cloning the LibreNMS source code with git ==== LibreNMS is installed using git. If you're not familiar with git, check out the git book or the tips at git ready. The initial install from github.com is called a '''git clone'''; subsequent updates are done through ''git pull''. The initial clone can take quite a while (nearly 3 minutes on a 10Mbps connection is typical) as the size of the software repository is 220+ MB in size. Run the following: {{{ # cd /opt # git clone https://github.com/librenms/librenms.git librenms }}} At this point, you should have a ''librenms'' directory, with the most recent revision checked out. ==== Web Interface ==== To prepare the web interface (and adding devices shortly), you'll need to create and change the ownership of a directory as well as create an Apache Virtul Host definition. First, create and chown the ''rrd'' directory and create the ''logs'' directory: {{{ # cd /opt/librenms # mkdir rrd logs # chown -R librenms:librenms /opt/librenms # chmod 775 rrd # chown www-data /opt/librenms }}} Next, create '''/etc/apache2/sites-available/librenms.conf''': {{{ # vi /etc/apache2/sites-available/librenms.conf }}} Add the following lines: {{{ DocumentRoot /opt/librenms/html/ ServerName librenms."your domain".ws.learn.ac.lk CustomLog /opt/librenms/logs/access_log combined ErrorLog /opt/librenms/logs/error_log AllowEncodedSlashes NoDecode Require all granted AllowOverride All Options FollowSymLinks MultiViews }}} Now enable the Virtual Host, but wait to restart Apache {{{ # a2ensite librenms.conf # a2enmod rewrite # service apache2 restart }}} ==== Web installer ==== You can choose either a web configuration or manual configuration at the command line. We're going to use the Web installer, which is by far the easiest, but we'll include the manual configuration as a reference at the end of this document. At this stage you can launch the web installer by going to http://librenms.'yourdomain'.ws.learn.ac.lk/install.php Follow the onscreen instructions. - Stage 0 is a summary of the PHP modules installed, normally you should just click on ''Next Stage'' - Stage 1 prompts you for the database settings. Enter * DB Host: localhost * DB User: librenms * DB Pass: * DB Name: librenms - Stage 2 is the DB creating itself - it should finish correctly, and you simply click on ''Goto Add User'' at the bottom - Stage 3: enter a username, password (do not forget the password) and E-mail address. This will become the login you use to access the web interface. - Stage 4 should show you the successful user creation, click on ''Generate Config'' - Stage 5: the interface should show, at this point: The config file has been created You can now click '''Finish install''' - Stage 6: you are done! You can now follow the instructions and click where it says '''click here to login to your new install'''. A useful tool is provided with LibreNMS to help verify that the software is installed correctly. Let's try it out: {{{ # cd /opt/librenms # ./validate.php }}} You may see warnings about the software not being up to date, and some more about permissions. You can probably ignore these for now, but it might come in useful later if you experience issues with LibreNMS. We can now secure the '''/opt/librenms''' directory again: {{{ # chown librenms /opt/librenms }}} === Configuring LibreNMS === ==== Setting the SNMP community ==== First, edit the file '''/opt/librenms/config.php''', # vi /opt/librenms/config.php and find the line: {{{ $config['snmp']['community'] = array("public"); }}} And change it to: {{{ $config['snmp']['community'] = "NetManage"; }}} ==== Tell LibreNMS which subnets it's allowed to scan automatically ==== By default, LibreNMS will try ask for the list of “neighbors” that network devices "see" on the network. This is done using the Link Layer Discovery Protocol (LLDP) or Cisco's CDP (Cisco Discovery Protocol). But to be on the safe side, and not scan networks outside your organization, LibreNMS needs to be told which subnets it's allowed to scan for new devices. Still in the file '''/opt/librenms/config.php''', find the line: {{{ #$config['nets'][] = "10.0.0.0/8"; }}} And replace this with the following to scan our specific subnets in use by our network and the workshop infrastructure. {{{ $config['nets'][] = "192.248.6.0/24"; }}} We need to make one more change... ==== Tell LibreNMS not to add duplicate devices ==== A situation can happen where two devices have duplicate SNMP sysName. (that's hostname in IOS) They could be two different devices, so it would be a good idea to have LibreNMS automatically add and monitor them. But it can also happen that the SAME device is seen multiple times by LibreNMS - once using LLDP/CDP, and another time via OSPF (for example). In that case, it ends up added twice. For instance, you may suddenly see two devices called rtr2-fa0-0.ws.ac.lk and rtr2, and this is not what we want. Since "both" devices are in fact the same, their SNMP sysName will be identical, and we can tell LibreNMS to NOT add devices if one already exists with the same sysName - after all, this shouldn't happen in a well configured network! To avoid this, add the following line at the bottom of the '''config.php''' file: {{{ $config['allow_duplicate_sysName'] = false; }}} ... this will prevent LibreNMS from adding the device if it exists already with the same sysName. You will be able to see if there are duplicate devices deteced in the Event Log (Overview -> Event Log). After you've added the above setting, save the file and exit - we’re nearly done! ==== Add a host ==== Let's add localhost (i.e.: YOUR virtual server), using the following commands. Later you'll do this from the Web interface: {{{ # cd /opt/librenms # php addhost.php localhost NetManage v2c }}} You should see: Added device localhost (1) Notice we explicitly tell LibreNMS which SNMP community to use. We also assume it's SNMP v2c. If you're using v3, there are additional steps which aren't provided here. ==== Discover and Poll newly added hosts ==== LibreNMS first “discovers” each host that has been added. This means that it methodically examines each host you added and figures out what it should monitor. The discover.php script does not automatically scan your network to find new devices. To run this script do: {{{ # cd /opt/librenms # sudo -u librenms php discovery.php -h all }}} NOTE: This could take some time. If you try to add devices that do not yet have an snmp service configured, then the discovery script takes a while to time out. Once this has finished you can now "poll" the hosts. This means LibreNMS now knows what it wishes to monitor for each host, but it has yet to populate its database with initial values for each item. To do this we do: {{{ # sudo -u librenms php poller.php -h all }}} As you can see the poller.php script does quite a bit with just a few devices. When we add it to a cronjob below this helps explain why LibreNMS is a resource intensive tool. ==== Create cronjob ==== Create the cronjob which will run periodic tasks required by LibreNMS: {{{ # cd /opt/librenms # cp librenms.nonroot.cron /etc/cron.d/librenms }}} ==== Install complete ==== That's it! You now should be able to log in to http://librenms.'your domain'.ws.learn.ac.lk/ and begin to explore the information being collected for your monitored devices. PLEASE NOTE: We have not covered HTTPS setup in this example, so your LibreNMS install is not secure by default. Please do not expose it to the public Internet unless you have configured HTTPS and taken appropriate web server hardening steps.