VPN Life: Servers - Keep Them Hard

As I mentioned in my last post, we’re not quite at the point of installing the VPN. There are a few leftover tasks before our servers are hard enough, so let’s finish them off…

Note: I’ve really been trying to avoid any ‘in the gutter’ remarks - but it’s getting more difficult by the week…

Automating Our Rootkit Checkers

Last time, I showed this snippet which installed two rootkit checkers and then ran them once.

apt-get install -y rkhunter chkrootkit
chkrootkit
rkhunter --update
rkhunter --propupd
rkhunter --check

While that’s nice, in reality, these should be run weekly, if not nightly. So, let’s automate them.

Unsurprisingly, I’m just going to use a cronjob here, as it’s good enough and simple. Let’s set up the rootkit checkers to run in the morning, one at 3am and one at 4am. Note: I had this as a one-line command, but for some reason it didn’t work reliably:

crontab -l > tmpcron
echo "0 3 * * * chkrootkit" >> tmpcron
echo "0 4 * * * /usr/bin/rkhunter --cronjob --update --quiet" >> tmpcron
crontab tmpcron
rm tmpcron

I’ve not included it, but these checkers on their own are nice I guess, but meh… What you should be doing is setting up your servers to send outgoing mail and then add the appropriate mailing commands to these cronjobs (I’m leaving it out, because Mail is always a can of worms for me… However, use SendGrid - be happy).

Get Your Blanket

One of the final things we did last time was to run Lynis as a security audit. After the work done last time, we still only ended up with around a score of 70/100. The number was actually a 68 and it’s broken down below (note: your number may be different depending on what you’ve installed on your machine - because Lynis is an opportunistic scanner which reacts to installed components).

================================================================================

  -[ Lynis 2.2.0 Results ]-

  No warnings

  Suggestions (28):
  ----------------------------
  - Set a password on GRUB bootloader to prevent altering boot configuration (e.g. boot in single user mode without password) [BOOT-5122]
      https://cisofy.com/controls/BOOT-5122/
  - Install a PAM module for password strength testing like pam_cracklib or pam_passwdqc [AUTH-9262]
      https://cisofy.com/controls/AUTH-9262/
  - Configure minimum password age in /etc/login.defs [AUTH-9286]
      https://cisofy.com/controls/AUTH-9286/
  - Configure maximum password age in /etc/login.defs [AUTH-9286]
      https://cisofy.com/controls/AUTH-9286/
  - Default umask in /etc/login.defs could be more strict like 027 [AUTH-9328]
      https://cisofy.com/controls/AUTH-9328/
  - Default umask in /etc/init.d/rc could be more strict like 027 [AUTH-9328]
      https://cisofy.com/controls/AUTH-9328/
  - To decrease the impact of a full /home file system, place /home on a separated partition [FILE-6310]
      https://cisofy.com/controls/FILE-6310/
  - To decrease the impact of a full /tmp file system, place /tmp on a separated partition [FILE-6310]
      https://cisofy.com/controls/FILE-6310/
  - Check your /etc/fstab file for swap partition mount options [FILE-6336]
      https://cisofy.com/controls/FILE-6336/
  - Disable drivers like USB storage when not used, to prevent unauthorized storage or data theft [STRG-1840]
      https://cisofy.com/controls/STRG-1840/
  - Install debsums utility for the verification of packages with known good database. [PKGS-7370]
      https://cisofy.com/controls/PKGS-7370/
  - Install package apt-show-versions for patch management purposes [PKGS-7394]
      https://cisofy.com/controls/PKGS-7394/
  - Install ARP monitoring software like arpwatch [NETW-3032]
      https://cisofy.com/controls/NETW-3032/
  - Configure a firewall/packet filter to filter incoming and outgoing traffic [FIRE-4590]
      https://cisofy.com/controls/FIRE-4590/
  - Consider hardening of SSH configuration [SSH-7408]
    - Details: LogLevel (INFO --> VERBOSE)
      https://cisofy.com/controls/SSH-7408/
  - Consider hardening SSH configuration [SSH-7408]
    - Details: Port (22 --> )
      https://cisofy.com/controls/SSH-7408/
  - Consider hardening SSH configuration [SSH-7408]
    - Details: TCPKeepAlive (YES --> NO)
      https://cisofy.com/controls/SSH-7408/
  - Consider hardening of SSH configuration [SSH-7408]
    - Details: UsePrivilegeSeparation (YES --> SANDBOX)
      https://cisofy.com/controls/SSH-7408/
  - Consider hardening SSH configuration [SSH-7408]
    - Details: X11Forwarding (YES --> NO)
      https://cisofy.com/controls/SSH-7408/
  - Add a legal banner to /etc/issue, to warn unauthorized users [BANN-7126]
      https://cisofy.com/controls/BANN-7126/
  - Add legal banner to /etc/issue.net, to warn unauthorized users [BANN-7130]
      https://cisofy.com/controls/BANN-7130/
  - Enable process accounting [ACCT-9622]
      https://cisofy.com/controls/ACCT-9622/
  - Enable sysstat to collect accounting (no results) [ACCT-9626]
      https://cisofy.com/controls/ACCT-9626/
  - Enable auditd to collect audit information [ACCT-9628]
      https://cisofy.com/controls/ACCT-9628/
  - Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350]
      https://cisofy.com/controls/FINT-4350/
  - Determine if automation tools are present for system management [TOOL-5002]
      https://cisofy.com/controls/TOOL-5002/
  - One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000]
      https://cisofy.com/controls/KRNL-6000/
  - Harden compilers like restricting access to root user only [HRDN-7222]
      https://cisofy.com/controls/HRDN-7222/

  Follow-up:
  ----------------------------
  - Check the logfile for more details (less /var/log/lynis.log)
  - Read security controls texts (https://cisofy.com)
  - Use --upload to upload data (Lynis Enterprise users)

================================================================================

  Lynis security scan details:

  Hardening index : 68 [#############       ]
  Tests performed : 189
  Plugins enabled : 0

  Quick overview:
  - Firewall [X] - Malware scanner [V]

  Lynis Modules:
  - Compliance Status   [NA]
  - Security Audit      [V]
  - Vulnerability Scan  [V]

  Files:
  - Test and debug information      : /var/log/lynis.log
  - Report data                     : /var/log/lynis-report.dat

================================================================================

Lynis gives us a report with a score, the number of tests performed, a high level breakdown, but most importantly, a list of suggestions on what to improve…

So, let’s do that.

Below is a set of commands which will fix many of the Lynis suggestions.

## Install ARP monitoring software like arpwatch [NETW-3032]
sed -i "s/#eth0\t-m root/eth0\t-m root/" /etc/arpwatch.conf
/etc/init.d/arpwatch restart

## Install package apt-show-versions for patch management purposes [PKGS-7394]
apt-get install -y apt-show-versions

## Install debsums utility for the verification of packages with known good database. [PKGS-7370]
apt-get install -y debsums

## Install a PAM module for password strength testing like pam_cracklib or pam_passwdqc [AUTH-9262]
apt-get install libpam-passwdqc

## Configure minimum password age in /etc/login.defs [AUTH-9286]
## Configure maximum password age in /etc/login.defs [AUTH-9286]
sed -i "s/PASS_MAX_DAYS\t99999/PASS_MAX_DAYS\t3650/" /etc/login.defs
sed -i "s/PASS_MIN_DAYS\t0/PASS_MIN_DAYS\t1/" /etc/login.defs

## Consider hardening of SSH configuration [SSH-7408]
sed -i "s/LogLevel INFO/LogLevel VERBOSE/" /etc/ssh/sshd_config

## Consider hardening SSH configuration [SSH-7408]
PORT=pick-a-port-number
sed -i "s/Port 22/Port $PORT/" /etc/ssh/sshd_config

## Configure a firewall/packet filter to filter incoming and outgoing traffic [FIRE-4590]
apt-get install ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow $PORT/tcp
ufw enable

## Consider hardening SSH configuration [SSH-7408]
sed -i "s/TCPKeepAlive yes/TCPKeepAlive no/" /etc/ssh/sshd_config

## Consider hardening SSH configuration [SSH-7408]
sed -i "s/X11Forwarding yes/X11Forwarding no/" /etc/ssh/sshd_config

## Consider hardening of SSH configuration [SSH-7408]
sed -i "s/UsePrivilegeSeparation yes/UsePrivilegeSeparation sandbox/" /etc/ssh/sshd_config

## Disable drivers like USB storage when not used, to prevent unauthorized storage or data theft [STRG-1840]
echo "blacklist usb-storage" >> /etc/modprobe.d/blacklist.conf
update-initramfs -u

## Default umask in /etc/login.defs could be more strict like 027 [AUTH-9328]
sed -i "s/UMASK\t\t022/UMASK\t\t027/" /etc/login.defs

## Default umask in /etc/init.d/rc could be more strict like 027 [AUTH-9328]
sed -i "s/umask 022/umask 027/" /etc/init.d/rc

## Add a legal banner to /etc/issue, to warn unauthorized users [BANN-7126]
echo "Warning! Unauthorized access to this server is expressly prohibited." >> /etc/issue

## Add legal banner to /etc/issue.net, to warn unauthorized users [BANN-7130]
echo "Warning! Unauthorized access to this server is expressly prohibited." >> /etc/issue.net

## Enable sysstat to collect accounting (no results) [ACCT-9626]
apt-get install sysstat
sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat

## Check what deleted files are still in use and why. [LOGG-2190]
## To find the files, run this 'lsof -nP +L1' - In reality, a reboot will fix this

## Purge old/removed packages with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. [PKGS-7346]
dpkg --list | grep "^rc" | cut -d " " -f 3 | xargs sudo dpkg --purge

So, run these and then re-run Lynis, and VOILA12T10 We’re at a gentlemanly 85/100.

================================================================================

  -[ Lynis 2.2.0 Results ]-

  No warnings

  Suggestions (11):
  ----------------------------
  - Set a password on GRUB bootloader to prevent altering boot configuration (e.g. boot in single user mode without password) [BOOT-5122]
      https://cisofy.com/controls/BOOT-5122/
  - To decrease the impact of a full /home file system, place /home on a separated partition [FILE-6310]
      https://cisofy.com/controls/FILE-6310/
  - To decrease the impact of a full /tmp file system, place /tmp on a separated partition [FILE-6310]
      https://cisofy.com/controls/FILE-6310/
  - Check your /etc/fstab file for swap partition mount options [FILE-6336]
      https://cisofy.com/controls/FILE-6336/
  - Check iptables rules to see which rules are currently not used [FIRE-4513]
      https://cisofy.com/controls/FIRE-4513/
  - Enable process accounting [ACCT-9622]
      https://cisofy.com/controls/ACCT-9622/
  - Enable auditd to collect audit information [ACCT-9628]
      https://cisofy.com/controls/ACCT-9628/
  - Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350]
      https://cisofy.com/controls/FINT-4350/
  - Determine if automation tools are present for system management [TOOL-5002]
      https://cisofy.com/controls/TOOL-5002/
  - One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000]
      https://cisofy.com/controls/KRNL-6000/
  - Harden compilers like restricting access to root user only [HRDN-7222]
      https://cisofy.com/controls/HRDN-7222/

  Follow-up:
  ----------------------------
  - Check the logfile for more details (less /var/log/lynis.log)
  - Read security controls texts (https://cisofy.com)
  - Use --upload to upload data (Lynis Enterprise users)

================================================================================

  Lynis security scan details:

  Hardening index : 85 [#################   ]
  Tests performed : 192
  Plugins enabled : 0

  Quick overview:
  - Firewall [V] - Malware scanner [V]

  Lynis Modules:
  - Compliance Status   [NA]
  - Security Audit      [V]
  - Vulnerability Scan  [V]

  Files:
  - Test and debug information      : /var/log/lynis.log
  - Report data                     : /var/log/lynis-report.dat

================================================================================

Shallow Dives

There are a few items which kinda threw me off. I guess I see the value, but they seem superficial.

Configure maximum password age in /etc/login.defs [AUTH-9286] - The max age only needs to be NOT the default and this test passes. I set it to 10 years and it passed… I would have preferred if there was a recommended protocol like every year, 60 days, etc. I know that’s a hard request, but passing me with a 10 year password is kinda rough. I suppose the key for Lynis is just bringing up the fact that there is a poor default, and it’s up to the sysadmin to fix that as they/the company sees fit.

Consider hardening of SSH configuration [SSH-7408] -> Details: LogLevel (INFO –> VERBOSE) - I don’t really see how changing the LogLevel from INFO to DEBUG hardened SSH.

Consider hardening SSH configuration [SSH-7408] -> Details: Port (22 –> ) - This task required changing the default port number away from 22. This is one I’ve debated a lot, and you’ll notice that in the last post, I didn’t change it by default. I used to always use a non-default SSH port, but then didn’t see much value in it since a port scanner can always find the SSH port relatively quickly. The only value I see is that it helps stop robots who are just doing drive-by brute force attacks without a port scan… The downside is that it’s security through obscurity, which I’m not a big fan of.

Add a legal banner to /etc/issue, to warn unauthorized users [BANN-7126], Add legal banner to /etc/issue.net, to warn unauthorized users [BANN-7130] - I really hope these “add a legal banner” tests were weighted very low. Looking through Lynis’s code, as I can see, this test just looks for any of the following words in the text: “access authorized legal monitor owner policy policies private prohibited restricted this unauthorized criminal evidence expressly authority warning”

Perfection Isn’t Worth It

The first thing to notice from the before/after Lynis results is that I didn’t get to 100%, in fact, I didn’t even make it to 90%!

Right off the bat, there are a few tests which are kinda fluffy - such as Check your /etc/fstab file for swap partition mount options [FILE-6336] and Check iptables rules to see which rules are currently not used [FIRE-4513] which tell you to check something, but don’t give immediately actionable tasks.

With some of the auditing tasks, they’re again kind of loose, as they suggest to install auditing tools - but not to check what they should audit, so they’re freebies that I don’t think should be done (I’m never going to check those audit logs). Also, with some of the auditing and reporting tools, they use those tests to upsell Lynis Enterprise (e.g. “for auditing and reporting settings, you should upgrade to Lynis Enterprise”). I don’t hold anything against them for this, as they’ve provided a GREAT product for free already.

Finally, there are tasks regarding splitting up certain directories into other partitions - which I do see value in, and will probably do at some point, just don’t feel like doing it right now.

What About DenyHosts/Fail2Ban?

DenyHosts and Fail2Ban are automated daemons which protect servers by detecting malicious traffic in the logs and modify your IPTables to block access attempts from those IP addresses.

For example, if someone tries to ssh into your servers from x.y.z.a and fails to login 3 times, x.y.z.a is automatically banned for some amount of time (which you control).

I was originally going to write about setting these tools up (or at least, setting one of them up), but I’m going to hold off for now, and I might update this later. The last time I set these guys up, I somehow locked myself out of my own servers (I might have not been using SSH keys at the time).

In any case, here are some Digital Ocean tutorials (here and here) on how to set them up, if you’re interested. They do roughly the same thing (modify IPTables based on logs), so I would only set up one of them.

Next Week