Changes between Version 1 and Version 2 of NspwUprouse/Agenda/netflow

Timestamp:

Sep 12, 2021, 9:10:43 PM (3 years ago)

Author:

admin

Comment:

--

NspwUprouse/Agenda/netflow

@@ -8 +8 @@
 {{{
 $ sudo apt-get install build-essential autoconf
-$ sudo apt-get install rrdtool mrtg librrds-perl librrdp-perl librrd-dev \
-       libmailtools-perl bison flex
+$ sudo apt-get install rrdtool mrtg librrds-perl librrdp-perl librrd-dev libmailtools-perl bison flex libbz2-dev libclang-dev
 }}}
 
@@ -16 +15 @@
 Now proceed to download and build. Note that only the last step (make install) has to be done as root.
 {{{
-$ cd
-$ wget http://192.248.4.49/NetMon/nfdump-1.6.13.tar.gz
-$ tar xvzf nfdump-1.6.13.tar.gz
-$ cd nfdump-1.6.13
-$ ./configure --help      # optional, shows the build settings available
+$ cd ~ 
+$ git clone https://github.com/phaag/nfdump.git
+$ cd nfdump
+$ ./autogen.sh
 $ ./configure --enable-nfprofile --enable-nftrack --enable-sflow
 $ make
 $ sudo make install
+$ sudo ldconfig
 }}}
 
 == Router Configuration ==
 
-'''Note: This part is already done for you.'''
+
 
 === Cisco Router ===
@@ -127 +126 @@
 Process the file(s) with nfdump:
 {{{
-nfdump -r /tmp/nfcap-test/nfcapd.201Ywwxxyyzz | less
-nfdump -r /tmp/nfcap-test/nfcapd.201Ywwxxyyzz -s srcip/bytes
+nfdump -r /tmp/nfcap-test/nfcapd.202Ywwxxyyzz | less
+nfdump -r /tmp/nfcap-test/nfcapd.202Ywwxxyyzz -s srcip/bytes
 }}}
 w,x,y,z indicate year, month, day and time.
@@ -150 +149 @@
 
 {{{
-nfdump -r /tmp/sfcap-test/nfcapd.201Ywwxxyyzz | less
-nfdump -r /tmp/sfcap-test/nfcapd.201Ywwxxyyzz -s srcip/bytes
+nfdump -r /tmp/sfcap-test/nfcapd.202Ywwxxyyzz | less
+nfdump -r /tmp/sfcap-test/nfcapd.202Ywwxxyyzz -s srcip/bytes
 }}}
 w,x,y,z indicate year, month, day and time.
 
+== Set up folders and nfcapd ==
+
+Following commands are executed as root. You may use `sudo su` to become root.
+
+
+Create folders per device. 
+{{{
+mkdir -p /var/nfdump/profiles-data/live/source1/
+mkdir -p /var/nfdump/profiles-data/live/source2/
+}}}
+In the above, a device has been named as a source.
+
+Provide access to the apache2 user:
+
+{{{
+chown -R www-data:www-data /var/nfdump/profiles-data
+}}}
+
+Assuming device source1 is a netflow device and device source2 is a sflow device, run the following to start the flow collectors.
+
+{{{
+/usr/local/bin/nfcapd -w -D -p 9995 -u www-data -g www-data -B 200000 -S 1 -z -I source1 -l /var/nfdump/profiles-data/live/source1/
+/usr/local/bin/sfcapd -w -D -p 9996 -u www-data -g www-data -B 200000 -S 1 -z -I source2 -l /var/nfdump/profiles-data/live/source2/
+}}}
+
+You can check whether the services are running by observing `netstat -nlp` for open udp port numbers. If they are not working, tail the `/var/log/syslog` for possible errors.
+
 == Installing and setting up NfSen ==
 
 Download and compile nfsen.
 {{{
-$ cd
-$ wget http://wget http://192.248.4.49/NetMon/nfsen-1.3.6p1.tar.gz
-$ tar xvzf nfsen-1.3.6p1.tar.gz
-$ cd nfsen-1.3.6p1
-$ perl -MCPAN -e 'install Socket6'
-Would you like to configure as much as possible automatically? [yes]
-What approach do you want? (Choose local::lib, sudo or manual)
-[local::lib]
-$ cd etc
-$ cp nfsen-dist.conf nfsen.conf
-$ vi nfsen.conf
-}}}
-
-Set the $BASEDIR variable
-{{{
-$BASEDIR = "/var/nfsen";
-}}}
-
-Set the users appropriately so that Apache can access files:
-{{{
-$WWWUSER = 'www-data';
-$WWWGROUP = 'www-data';
-}}}
-
-Set the buffer size to something small, so that we see data quickly. You would not do this on a production system.
-
-# Receive buffer size for nfcapd - see man page nfcapd(1)
-{{{
-$BUFFLEN = 2000;
-}}}
-
-Find the %sources definition, and change it to:
-
-'''For a netflow router'''
-{{{
-%sources=(
-'accessrtr' => {'port'=>'9001','col'=>'#0000ff','type'=>'netflow'},
- );                
-}}}
-
-'''For a sflow router'''
-{{{
-%sources=(
-'accessrtr' => {'port'=>'9001','col'=>'#0000ff','type'=>'sflow'},
- );                
-}}}
-
-(substitute your group's router for accessrtr, and either remove or comment out the existing sample sources).
-
-Change the HTMLDIR from /var/www/nfsen/ to /var/www/html/nfsen/
-{{{
-$HTMLDIR    = "/var/www/html/nfsen/";
-}}}
-Now save and exit from the file.
-
-The default rrd tool version for nfsen is 1.5 but the latest version 1.6 therefore a slight configuration is needed
-{{{
-cd
-vi nfsen-1.3.6p1/libexec/NfSenRRD.pm
-}}}
-
-Find the following line'
-{{{
-if ( $rrd_version >= 1.2 && $rrd_version < 1.5 ) 
-}}}
-
-change it to,
-{{{
-if ( $rrd_version >= 1.2 && $rrd_version < 1.6 ) 
-}}}
-
-Create the netflow user on the system
-{{{
-$ sudo useradd -d /var/nfsen -G www-data -m -s /bin/false netflow
-}}}
-
-Install NfSen and start it
-
-Change directory back to just inside the source directory:
-{{{
-$ cd
-$ cd nfsen-1.3.6p1
-}}}
-
-Now, finally, we install:
-{{{
-$ sudo perl install.pl etc/nfsen.conf
-}}}
-
-Press ENTER when prompted for the path to Perl.
-
-Install init script
-
-In order to have nfsen start and stop automatically when the system starts, add a link to the init.d directory pointing to the nfsen startup script:
-{{{
-$ sudo ln -s /var/nfsen/bin/nfsen /etc/init.d/nfsen
-$ sudo update-rc.d nfsen defaults 20
-}}}
-
-Start NfSen
-{{{
-$ sudo service nfsen start
-}}}
+# run following commands as root
+# install packages
+apt install apache2 git nfdump pkg-config php7.4 php7.4-dev libapache2-mod-php7.4 rrdtool librrd-dev
+# enable apache modules
+a2enmod rewrite deflate headers expires
+# install rrd library for php
+pecl install rrd 
+# create rrd library mod entry for php
+echo "extension=rrd.so" > /etc/php/7.4/mods-available/rrd.ini
+# enable php mod
+phpenmod rrd
+# configure virtual host to read .htaccess files
+vi /etc/apache2/apache2.conf # set AllowOverride All for /var/www
+# restart apache web server
+systemctl restart apache2
+# install nfsen-ng
+cd /var/www/html # or wherever
+git clone https://github.com/mbolli/nfsen-ng
+chown -R www-data:www-data .
+chmod +x nfsen-ng/backend/cli.php
+# next step: configuration
+}}}
+
+Create the settings file for nfsen-ng
+
+{{{
+cp /var/www/html/nfsen-ng/backend/settings/settings.php.dist /var/www/html/nfsen-ng/backend/settings/settings.php
+}}}
+
+Edit the settings.php file and include the device names by '''editing''' the following lines.
+
+For the sources:
+{{{
+        'sources' => array(
+            'source1', 'source2',
+}}}
+
+For the nfdump:
+{{{
+    'nfdump' => array(
+        'binary' => '/usr/bin/nfdump',
+        'profiles-data' => '/var/nfdump/profiles-data',
+        'profile' => 'live',
+        'max-processes' => 1, // maximum number of concurrently running nfdump processes
+    ),
+}}}
+
+Next, import existing data:
+{{{
+/var/www/html/nfsen-ng/backend/cli.php  import
+}}}
+
+Now we can start the nfsen-ng daemon:
+{{{
+/var/www/html/nfsen-ng/backend/cli.php  start
+}}}
+
 
 View flows via the web:
 
-You can find the nfsen page here:
-
-'''http://<your IP address>/nfsen/nfsen.php
-
-You may see a message such as:
-
-'''Frontend - Backend version mismatch!'''
-
-This will go away if you reload the page, it's not a problem.
-
-== Using NfSen to identify top talkers ==
-
-Now let's use NfSen to explore the traffic flows in the network, with the aim of finding out who was been downloading the most data. Look carefully at the output generated at each step - ask an instructor to explain if you don't understand what you see.
-
- - Navigate to the Detail page
-
- - Select the time window. to do that change from "Single Timeslot" to '''Time Window''. Once you have done this, the vertical selector arrow and line in the graph window can be split.
-
-[[Image(https://ws.learn.ac.lk/raw-attachment/wiki/netmon2017netflow/ns1.png)]]
-
- - Pull the left half of the arrow to the left and the right half to the right, to select the time period of interest. Then you should see some summary statistics appear in the table below the graph, for the time period you have selected
-
-[[Image(https://ws.learn.ac.lk/raw-attachment/wiki/netmon2017netflow/ns2.png)]]
-
- - List individual flows by Selecting "List Flows", make sure none of the "Aggregate" boxes are checked, and then click process. This will display some flows at the beginning of the time period. Click '''process'''. You will see the top flows below.
-
-[[Image(https://ws.learn.ac.lk/raw-attachment/wiki/netmon2017netflow/ns3.png)]]
-
- - By selecting "bi-directional" you can get NfSen to associate the inbound and outbound flows into a single line
-
- - If we know which host we want to examine, we can apply a filter to show only those flows to and from that host. Do this by entering "host x.x.x.x" in the filter box, and then pressing the process again. (Replace x.x.x.x with the address of one of the host PC)
-
-[[Image(https://ws.learn.ac.lk/raw-attachment/wiki/netmon2017netflow/ns4.png)]]
-
- - The next thing we can do is to get NfSen to sort the flows by a number of bytes. Remove any filter from the Filter box; select "Stat TopN", stat "Flow Records", order by "Bytes". Ensure all the aggregate boxes are unchecked, then press the process
-
-[[Image(https://ws.learn.ac.lk/raw-attachment/wiki/netmon2017netflow/ns5.png)]]
- 
- - NFsen can show you inbound traffic grouped by receiver IP address. which means showing the total amount of traffic delivered to that host. To do this, Stat "DST IP Address", order by "bytes". Then apply a filter that shows only traffic to your group's network: "dst net 192.248.6.0/24". You can do the same with a single host.
-
-[[Image(https://ws.learn.ac.lk/raw-attachment/wiki/netmon2017netflow/ns6.png)]]
-
- - By clicking on an IP address, you will get some information from reverse DNS and whois.
+You can find the nfsen-ng page here:
+
+'''http://<your IP address>/nfsen-ng/frontend/'''
+
+In case of server restart, you need to re-run,
+
+{{{
+/usr/local/bin/nfcapd -w -D -p 9995 -u www-data -g www-data -B 200000 -S 1 -z -I source1 -l /var/nfdump/profiles-data/live/source1/
+/usr/local/bin/sfcapd -w -D -p 9996 -u www-data -g www-data -B 200000 -S 1 -z -I source2 -l /var/nfdump/profiles-data/live/source2/
+/var/www/html/nfsen-ng/backend/cli.php  start
+}}} 
+
+You may also automate that by keeping a bash script to run on every reboot via cron jobs.
+
+
+== References: ===
+
+1. https://github.com/mbolli/nfsen-ng
+2. https://github.com/phaag/nfdump
+3. https://www.systutorials.com/docs/linux/man/1-nfcapd/
+