Velas Validators Security Recommendations

brought to you by Premagine/TrueStaking

Ode to Linux System Administration Generic Best Practices

You can find many good lists of generic best practices for Linux System Administrators to follow. You can find short lists such as this Cardano forum entry, or long and very detailed lists with tools for remediation such as the CIS security benchmarks. Here's a key point summary:

While instructions for all the above items should be quickly found on the Internet, if you have any questions feel free to reach out. @perltk (telegram)

note that backups, logging and monitoring, resiliency – are all related topics but not covered here.

note If you follow the remainder of this guide we will address two of the items above: the custom port for “Avoiding weak remote access” and the host firewall rules for “Control Incoming Network Connections”.

Ready to go beyond the cyber security basics?

Advanced Preventive Control: AppArmor

AppArmor is a kernel level mechanism to assign rights to a running process and restrict what files/directories the process can read/write/update. With the Velas validator service accepting connections for RPC calls (TCP Port 8899) and accepting inbound UDP connections on ports 8000–8010 – it makes sense that we should take the extra step to secure the service with Ubuntu AppArmor.

AppArmor is installed and loaded by default in modern Ubuntu. However, we will want to install the optional AppArmor utils package.

sudo apt install apparmor-utils

Next, we will will create an AppArmor profile for our velas-validator service. Profiles are simply text files stored in /etc/apparmor.d/ . The only trick is, the filename of the profile must match the full path and name of the executable where every “/” in the pathname becomes a “.” in the filename. If you followed the validator installation instructions here then your executable should be in $HOME/.local/share/velas/install/active_release/bin/velas-validator . Otherwise, you will find it on the “ExecStart” line of your /etc/systemd/system/velas.service file or wherever you moved it.

Now, convert the full path into a file name by replacing all “/” with “.” and /home/velasnode/.local/share/velas/install/active_release/bin/velas-validator – becomes home.velasnode..local.share.velas.install.active_release.bin.velas-validator (note the double “..” and that the leading “/” is ignored)

Armed with this knowledge, with the editor of your choice, create /etc/apparmor.d/full.path.to.velas-validator with the following content:

#include >tunables/global>
  /home/velasnode/.local/share/velas/install/active_release/bin/velas-validator flags=(complain) {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/openssl>
}

Note that we specified “complain” mode, and thus the system won't block any access but will allow and log all access attempts.

Make sure you replace the “/home/velasnode/.local/share/velas/install/active_release/bin/velas-validator” on line 2 above with the appropriate path in your setup

Now enable the profile

sudo aa-complain /full/path/to/velas-validator

If there are no errors, then let this run for a bit. You can see any logs in /var/log/syslog

Now we capture all the apparmor complaints and build the profile automatically:

sudo aa-logprof (Use “I” for inherit, “A” for allow, and “S” for save.)

Detect or Prevent? To finalize the AppArmor protections, you have to choose: Do you leave it in “complain” mode and use it for “detection” or do you switch into “enforce” mode and use it for “prevention”?

If you choose detection, then simply leave it in complain mode and use your favorite log monitoring tool to monitor and alert on apparmor=“ALLOWED” events in /var/log/syslog.

If you do not have such a log monitoring toolset in your toolbox, then jump ahead to the next section “Egress Anomaly Alerting”. You can use swatchdog and a simple alerting script to monitor for apparmor events as they arrive in /var/log/syslog.

If you choose to transition to enforcement mode, then you will want to stop/start the Velas service at this point, just to ensure that any process changes get written and read from /proc and find their way into the profile.

Now we capture all the apparmor complaints and build the profile automatically:

sudo aa-logprof (Use “I” for inherit, “A” for allow, and “S” for save.)

You are encouraged to manually review the profile in /etc/apparmor.d/full.path.to.velas-validator. Below is a fully functioning profile, but your pathnames might be different.

/usr/local/bin/velas-validator flags=(complain) {
#include <abstractions/base>
#include <abstractions/nameservice>
#include <abstractions/openssl>
/bin/tar mrix,
/data/ledger/** rw,
/lib/x86_64-linux-gnu/ld-*.so mr,
/proc/sys/kernel/random/uuid r,
/sys/devices/** r,
/sys/fs/cgroup/cpu,cpuacct/system.slice/** r,
/sys/kernel/mm/transparent_hugepage/enabled r,
/usr/local/bin/velas-validator mr,
owner /data/ledger/ r,
owner /data/ledger/** rlwk,
owner /proc/*/cgroup r,
owner /proc/*/mountinfo r,
owner /proc/*/task/** rw,
owner /sys/fs/cgroup/cpu,cpuacct/user.slice/*/*/cpu.cfs_quota_us r,
owner @{HOME}/velas/validator.log rw,
owner @{HOME}/velas/wallet/** r,
}

Now you can view the updated and finalized profile in /etc/apparmor.d/full.path.to.your.velas-validator executable

If all looks good, then flip to enforcement mode with:

sudo aa-enforce /full/path/to/velas-validator

Done.

Of course, if something breaks, just flip back to complain mode with sudo aa-complain /full/path/to/velas-validator

Ok, that wasn't all that difficult, and we can find most of those elements pretty easily on the web. The next control we do is more obscure but this is one of the single most effective detective tools to have enabled on your validator. We've never seen a hacked server yet where the hacker didn't at least try to fetch additional code, perform lateral recon, or try to phone home. Detecting network egress anomalies catches the bad guys everytime.

Advanced Detective Control: Egress Anomaly Alerting

Perequisites:

Skill Level: Intermediate

Conceptual Overview

A Velas validator should be a dedicated server. It is dedicated to the validation task and is not used for other unnecessary tasks such as running a web server, email gateways, gaming, etc… As a dedicated device, we can easily create a baseline of network egress and then alert on any anomalous events. This is a key detective control as any successful exploit of the server will need to download other code to establish persistence or exfil data.

1. We will use iptables, ipset, swatchdog, and some simple scripts to create the baseline.
2. With the baseline built, we will make the iptables, ipset, and swatchdog services persistent.
3. Lastly, we will enable alerting, if you have a commandline tool to accomplish this task
        Email -> SMS text is a popular option.  (What we use)
        SMS gateway services like [textbelt](https://textbelt.com/) are also easy to use
        Telegram is also a popular option , see https://github.com/vysheng/tg/wiki/Running-Telegram-CLI-as-Daemon

Step 1: Install ipset

sudo apt install ipset

Step 2: Create an iptables list to use as a baseline

sudo ipset create egress_seen iphash

Step 3: Install SwatchDog

We use swatchdog to monitor system messages and when it sees an alert from IPtables it calls the script to parse the log and add the IP address to the ipset list "egress_seen".

sudo apt install swatch

With the editor of your choice, put the following contents into ~/.swatchdogrc

watchfor /EGRESS_ALERT/
exec echo $_ | ~/egress_build.pl  

Step 4: Create Script to Build Baseline

This is the script we referenced under step 3 - just a simple PERL script to parse the log line and make a call to ipset to add the destination IP address associated with the new egress event.

With the editor of your choice, create ~/egress_build.pl

#!/usr/bin/perl  
$log=<STDIN>;  
chomp $log;
$log=~/.+DST=([0-9.]+).+PROTO=([A-Z]+).+DPT=([0-9]+)/;
$ip=$1;
$proto=$2;
$dport=$3;
system ("ipset add egress_seen $ip");  
system ("logger added $ip to ipset list egress_seen");

chmod +x ~/egress_build.pl

Step 5 : Disable UFW

If you followed the Velas Validator installation guide, then the following should be true:

You have tcp 8899 (RPC calls inbound)
You have udp 8000–8010 (validator traffic in/out)

And, if you followed the general linux security practices you already have your own customized SSH port (right? If not, this is a good time to make that change!)

IF you haven't already changed SSH to something other than port 22, do that now.

sudo nano /etc/ssh/sshd_config and change the port

(write down the port you just set – you will need it in the next step)

sudo ufw disable
sudo systemctl daemon-reload
sudo systemctl restart sshd

Now check sudo ufw status (It should be disabled? If so, continue to the next step)

Without closing your current terminal session to your validator, open another terminal session and ensure that you can SSH to your validator. This is just a failsafe, to double check that you changed the port successfully and you know what that new port is… if all is working as expected, then proceed to the next step.

Step 6: Deploy Velas IPtables

Note: this is based off the defaults in the Velas node build documentation.  **IF** you have changed the default port for SSH -- perhaps in step 5 above or as part of your own base build process -- you **MUST** change the line 9 below and replace “22” with your custom port for SSHD…

IF you have other services running on the box that require incoming connections, then you’ll need to add appropriate allow rules.
Using the editor of your choice, create ~/fw.sh with the following content:

iptables -F
iptables -X  
iptables -P INPUT DROP  
iptables -P OUTPUT ACCEPT  
iptables -P FORWARD ACCEPT  
iptables -I INPUT -i lo -j ACCEPT  
iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT  
iptables -A INPUT -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT  
iptables -I INPUT -p tcp --dport 22 -j ACCEPT  
iptables -I INPUT -p tcp --dport 8899 -j ACCEPT  
iptables -I INPUT -p udp --match multiport --dports 8000:8010 -j ACCEPT  
iptables -A OUTPUT -m set --match-set egress_seen dst -j ACCEPT  
iptables -A OUTPUT -o eth0 -m state --state NEW -j LOG --log-prefix "EGRESS_ALERT:" --log-level 6 --log-uid  

**DOUBLECHECK the port you have for line 9! If that isn't correct, you could deny yourself SSH access and have to use a rescue console access mechanism.

Now execute with:

sudo bash fw.sh

Step 7 : Build the Baseline

sudo swatchdog -t /var/log/syslog

(Note: by default, the config file is ~/.swatchdogrc, which we edited above already so we don't need to specify the config file name on the command line)
Let this run for a few minutes… then CTRL-C to close it.

Now check the list with

sudo ipset list egress_seen

You should see a number of IP addresses in the list – this is the list of current validators. Over a day or so of this building activity you would pick up your DNS servers, software update repositories, NTP time source, monitoring servers you send data to, etc…

To reset the baseline you can simply do: sudo ipset flush egress_seen

Step 8 : Enable Persistence

First, we make sure our ipset egress list survives a reboot

IF using Ubuntu 20+, then run the below and skip to the second step of this section to apply persistence to IPtables.

sudo apt install ipset-persistent

IF using Ubuntu 18.04 or older, we will manage persistency with a systemd service file. Create /etc/systemd/system/ipset-persistent.service with this content:

[Unit]  
Description=ipset persistent configuration  
Before=network.target  
Before=netfilter-persistent.service  
ConditionFileNotEmpty=/etc/iptables/ipsets>  

[Service]  
Type=oneshot  
RemainAfterExit=yes  
ExecStart=/sbin/ipset restore -exist -file /etc/iptables/ipsets  
ExecStop=/sbin/ipset save -file /etc/iptables/ipsets  
ExecStop=/sbin/ipset flush  
ExecStopPost=/sbin/ipset destroy  

[Install]
WantedBy=multi-user.target  
RequiredBy=netfilter-persistent.service  

If it doesn't already exist, then create /etc/iptables directory and save our ipset list

sudo /etc/iptables
ipset save -file /etc/iptables/ipsets

Second, we ensure our IPtables will survive a reboot

sudo apt install iptables-persistent netfilter-persistent

(Say “yes” to save the existing rules to disk. These will be located in /etc/iptables/rules.v4 and rules.v6 )

You can now run netfilter-persistent save to save the rules, and netfilter-persistent start to restore the rules from /etc/iptables/rules.v4. Note also, that you can edit the rules.v4 file directly and then simply use netfilter-persistent start to load the rules. Each line of the file is exactly like a command line entry simply without the leading “iptables ” command.

Third, we used swatchdog with a local config file and a local script to build our egress_seen ipset list

Now we need to a) transition from the commandline to a systemd service, b) use non-user config file, and c) also use a non-user script.

(a) Create /etc/systemd/system/swatchdog.service with the following content:

[Unit]  
Description=Swatchdog Service  
After=network.target  

[Service]  
Type=forking  
User=root  
ExecStart=/usr/bin/swatchdog -c /etc/swatchdog/swatchdog.conf -t '/var/log/syslog' --daemon  

[Install]  
WantedBy=multi-user.target

(b) Create /etc/swatchdog/swatchdog.conf

sudo mkdir /etc/swatchdog

With the editor of your choice create /etc/swatchdog/swatchdog.conf with the following content:

watchfor /EGRESS_ALERT/  
exec echo $_ /usr/local/bin/egress_alerts.pl  

(c) With the editor of your choice, create /usr/local/bin/egress_alerts.pl

#!/usr/bin/perl  
$log=<STDIN>;  
chomp $log;
$log=~/.+DST=([0-9.]+).+PROTO=([A-Z]+).+DPT=([0-9]+)/;
$ip=$1;
$proto=$2;
$dport=$3;
system ("ipset add egress_seen $ip");    
system ("/usr/local/bin/alert.sh $ip $proto $dport");

sudo chmod +x /usr/local/bin/egress_alerts.pl

(d) With the editor of your choice create /usr/local/bin/alert.sh. Here's what works for us:

#!/bin/bash  
mail -s "EGRESS ALERT" [email protected] <<< "Egress alert to $1,$2,$3"  

sudo chmod +x ~/usr/local/bin/alert.sh
systemctl enable netfilter-persistent
systemctl enable swatchdog
systemctl start swatchdog
systemctl enable ipset-persistent
systemctl start ipset-persistent
systemctl start netfilter-persistent

You now have an auto-updating egress baseline for your Velas Validator and enable real-time alarming in the event of an egress anomaly. Thanks for taking your Velas Validator cybersecurity posture seriously!

Recovery

If you followed the Velas Validator Build Guide, then you have a two identity keypairs: one for the validator identity and one for the vote account identity. Both of these identities are used constantly by the validation process and MUST be stored locally on the validator.

The primary focus of this document is cyber controls at the OS level of the validator. However, the #1 security task for a validator is not an OS level cyber control, but rather a procedural control. The most important security related task for a validator is to create an additional identity, set this new identity as the “vote-authorize-withdrawer” and then move it off the box into cold storage. The only time we routinely need it is to withdraw earnings from the vote account. The inconvenience of this pales in comparison to the value we achieve by being able to recover our validator in the event of a cyber intrusion. By separating the on-host validator identity and the withdrawer-authorized account, we deny the hacker access to our vote-account earnings and with this account we can recover and associate another validator identity with the vote account. Thus, with the withdrawer-authorized account in cold storage, we can effectively recover in the event our cyber controls fail.

The task is simple - just create a new identity:

velas-keygen new -o withdrawer-account-keypair.json
velas vote-authorize-withdrawer [VOTE-ACCOUNT-KEYPAIR.json][VALIDATOR-ACCOUNT-KEYPAIR.json] withdrawer-account-keypair.json

Now move the entire keypair to cold storage. There are many ways to achieve the value of “cold storage” – join the Velas Validators telegram group and share your ideas with the group.

Now you are prepared, even if our controls fail, we are in a position to recover.