Securing internet facing services

← Back

I've finally remembered the password from my blog! /s

For the past few months I've been fiddling with nginx and networking in hopes of making it as secure as I can, and I may have done a good job at it. Unfortunately, most blogs give you only bits and pieces, but I will try to provide the full picture with explanations. So, make yourself a cup of tea and let's dive right into it ☕

Please do note that this setup is the bare minimum for hosting public services.

Get an auto-renewable certificate

Prepare the base nginx config:

nano /etc/nginx/sites-available/example.com
server {
  listen         80;
  server_tokens  off;
  server_name    example www.example.com;  # Customize the target subdomains
  error_log      /var/log/nginx/example.error.log warn;  # Customize the error log location

  root           /path/to/static/files;  # Customize the static files' location
  index          index.html;
  try_files $uri /index.html;
}

Install certbot:

sudo apt update
sudo apt install -y certbot python3-certbot-nginx

Generate a certificate:

sudo certbot --nginx -d example.com -d www.example.com

Link the webpage to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/

Just a quick reference: unlink the webpage from the sites-enabled directory:

sudo unlink /etc/nginx/sites-enabled/example.com

Anubis

I have mixed feelings about using Cloudflare: on the one hand it saves from bots and DDOSes, on the other hand it's a single point of failure, that has been down a couple of times in recent years, and I want the website to be online as much as possible.

Anyways, since Cloudflare is banned in Russia, I don't have a choice but to use counterparts from local service providers or use third-party tooling; the choice is obvious 😉

Download the latest release:

Anubis releases

Install:

sudo apt update
sudo apt install curl
sudo apt install jq

arch=$(uname -m); case "$arch" in x86_64) a=amd64;; aarch64|arm64) a=arm64;; armv7l|armv7) a=armhf;; *) a="$arch";; esac
url=$(curl -sS https://api.github.com/repos/TecharoHQ/anubis/releases/latest
  | jq -r --arg a "$a" '.assets[] | select(.name | test(".deb$") and test($a; "i")) | .browser_download_url' | head -n1)
[ -z "$url" ] && echo "No .deb asset found for arch $arch (mapped: $a)" && exit 1
file=$(basename "$url")
curl -L --progress-bar -o "$file" "$url"
sudo apt update
sudo apt install -y ./"$file"

Copy the default policy:

sudo cp /usr/share/doc/anubis/botPolicies.yaml /etc/anubis/example.botPolicies.yaml

Copy the default config:

sudo cp /etc/anubis/default.env /etc/anubis/example.env

Edit the config:

nano /etc/anubis/example.env
BIND=:8923
DIFFICULTY=4
METRICS_BIND=:9090
POLICY_FNAME=/etc/anubis/example.botPolicies.yaml
SERVE_ROBOTS_TXT=0
TARGET=http://localhost:3923

Enable anubis:

sudo systemctl stop anubis@default.service
sudo systemctl disable anubis@default.service
sudo systemctl enable --now anubis@example.service

Nginx

sudo nano /etc/nginx/sites-available/example.com
upstream anubis {
    server 127.0.0.1:8923;  # the port should be the same as in BIND
    keepalive 30;
}

server {
    location / {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://anubis;
    }

    location ~ /.ht {
        deny all;
    }

    # certbot
}

server {
    listen 127.0.0.1:3923;  # The port should be the same as in TARGET
    # your server name
    # your site root
    # your site files
    # your site locations
    location / {
        port_in_redirect off;
    }
}

# http redirection
server {
    if ($host = www.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    server_name example.com www.example.com;
    return 404; # managed by Certbot
}

Nginx conf:

http {
    set_real_ip_from 127.0.0.1;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;
}

Enable http3

sudo nano /etc/nginx/sites-available/example.com
upstream anubis {
    server 127.0.0.1:8923;
    keepalive 30;
}

server {
    location / {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://anubis;
    }

    location ~ /.ht {
        deny all;
    }

    # certbot
    listen 443 ssl http2 fastopen=256; # managed by Certbot
    listen 443 quic reuseport;
    listen [::]:443 quic reuseport;

    add_header alt-svc 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000' always;

    http3 on;
    ssl_protocols TLSv1.3;
    quic_retry on;
    ssl_early_data on;
    proxy_set_header Early-Data $ssl_early_data;
    quic_gso on;

    ssl_stapling on;
    ssl_stapling_verify on;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header X-Frame-Options "DENY" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy strict-origin-when-cross-origin always;

    server_tokens off;
}

server {
    listen 127.0.0.1:3923;
    # your server name
    # your site root
    # your site files
    # your site locations
    location / {
        port_in_redirect off;
    }
}

# http redirection

Allowed methods

# GET - retrieve data
# HEAD - like a GET except that the response should not include the representation data in the body
# POST - process a resource in some way
# PUT - create or update a resource with the state in the request; A distinction from POST is that the client specifies the target location on the server.
# PATCH - modify a resource according to its partial state in the request; Compared to PUT, this can save bandwidth by sending only part of a resource's representation instead of all of it
# DELETE - delete a resource
# TRACE - respond with the received request in the response body; That way a client can see what (if any) changes or additions have been made by intermediaries.
# OPTIONS - report of the HTTP methods that are supported for a resource; can be used to check the functionality of a web server by requesting '*' instead of a specific resource
# CONNECT - requests that the intermediary establish a TCP/IP tunnel to the origin server identified by the request target. It is often used to secure connections through one or more HTTP proxies with TLS.

location / {
    if ($request_method !~ ^(GET|HEAD|POST)$ ) {
        return 444;
    }
    # ...
}

Vpn only access for a location

location ^~ / {
    allow xxx.xxx.xxx.xxx/32;  # vpn ip address
    deny all;                 # Deny all other IPs
}

If running a vpn on the same machine with nginx + anubis:

sudo nano /etc/nginx/nginx.conf
http {
    set_real_ip_from 127.0.0.1;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;
}
nano /etc/nginx/sites-available/example.com
location ^~ /admin.php {
    allow 127.0.0.1/32;
    deny all;
}
sudo nano /etc/sysctl.conf 
# https://wiki.archlinux.org/title/Sysctl# https://www.golinuxcloud.com/sysctl-config-for-high-performance-servers/
# Enable TCP Fast Open for both clients and the server
# https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html
# 0 - disabled
# 1 - enables sending data in the opening SYN on the client.
# 2 - enables the server support, i.e., allowing data in a SYN packet to be accepted and passed to the application before 3-way handshake finishes.
# 3 - enables the client and server support, so inbound and outbound# 1024 - enable all listeners to support Fast Open by default without explicit TCP_FASTOPEN socket option
net.ipv4.tcp_fastopen = 3
# Disable blackholling as a client
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 0
# Lowering this to 2 can help mitigate SYN flood attacks by reducing the length of time a socket is in SYN_RECV state, but this might impact clients with high latency or packet loss
net.ipv4.tcp_synack_retries = 2
# Protect against old duplicate packets disrupting a new connection's established state
net.ipv4.tcp_rfc1337 = 1
# Avoid a smurf attack
net.ipv4.icmp_echo_ignore_all = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Turn on protection for bad icmp error messages
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Turn on syncookies for SYN flood attack protection
net.ipv4.tcp_syncookies = 1
# Turn on and log spoofed, source routed, and redirect packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
# No source routed packets here
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Turn on reverse path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Make sure no one can alter the routing tables
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
# Don't act as a router
# If need to forward traffic - change the whole section from 0 to 1
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
# Enable ExecShield protection
# Red Hat only
kernel.exec-shield = 1
# Enable random placement of virtual memory regions protection
# Also randomize the heap
kernel.randomize_va_space = 2
# Tune IPv6
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Determines whether a system accepts prefix information in Router Advertisements
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
# Disables automatic generation of IPv6 addresses from Router Advertisement (RA) prefixes.
net.ipv6.conf.default.autoconf = 0
# Disable DAD in these scenarios:
# 1. Virtual machines or containers where address uniqueness is guaranteed by orchestration or configuration
# 2. Performance-critical environments where address assignment latency matters# 3. Loopback or internal interfaces with no risk of duplicates
net.ipv6.conf.default.dad_transmits = 0
# Maximum number of IPv6 addresses that can be autoconfigured on a network interface
net.ipv6.conf.default.max_addresses = 1
# Optimization for port use for LBs# Increase system file descriptor
limitfs.file-max = 65535
# Allow for more PIDs (to reduce rollover problems); may break some programs
kernel.pid_max = 65536
# Increase system IP port limits
net.ipv4.ip_local_port_range = 2000 65000
# Increase TCP max buffer size settable using setsockopt()
# https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/network_troubleshooting_and_performance_tuning/tuning-tcp-connections-for-high-throughput
# The first value is the minimum buffer size.
# The second value is the default buffer size.
# The third value is the maximum size of automatically tuned buffers.
# My connection
# 237.408ms or 0.237s
# 100 mbit
# (0.1 * 1000 * 1000 * 1000 / 8) * 0.237 = 2,962,500 bytes
# Max = 2×BDP ≈ 5.92 MiB = 6292500 bytes
# The second value should be less than 524288 bytes to prevent buffer collapsing
net.ipv4.tcp_rmem = 4096 262144 6292500
net.ipv4.tcp_wmem = 4096 262144 6292500
# Increase Linux auto tuning TCP buffer limits
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1
# sudo modprobe tcp_bbr
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Apply:

sudo sysctl -p /etc/sysctl.conf

Firewall

Please do note that ufw does NOT work with docker containers.

For it to work either install ufw-docker or configure iptables manually.

Install ufw:

sudo apt install ufw

Make sure IPv6 is enabled:

sudo cat /etc/default/ufwIPV6=yes

Forbid incoming connections:

sudo ufw default deny incoming

Forbid outgoing connections:

sudo ufw default deny outgoing

Allow SSH:

sudo ufw allow ssh

Rate limit SSH:

# if an IP address attempts to initiate six or more connections within 30 seconds, UFW will deny further connections from that IP
sudo ufw limit ssh

Check the rules:

Make sure that ssh is allowed!

sudo ufw show added

Enable ufw:

sudo ufw enable

Hardening PHP

sudo nano /etc/php/8.x/fpm/php.ini
; Disallow dangerous functions
disable_functions = phpinfo, system, mail, exec

; Try to limit resources

; Maximum execution time of each script, in seconds
max_execution_time = 30

; Maximum amount of time each script may spend parsing request data
max_input_time = 60

; Maximum amount of memory a script may consume (8MB)
memory_limit = 8M

; Maximum size of POST data that PHP will accept.
post_max_size = 8M

; Whether to allow HTTP file uploads.
file_uploads = Off

; Maximum allowed size for uploaded files.
upload_max_filesize = 2M

; Do not expose PHP error messages to external users
display_errors = Off

; Turn on safe mode
safe_mode = On

; Only allow access to executables in isolated directory
safe_mode_exec_dir = php-required-executables-path

; Limit external access to PHP environment
safe_mode_allowed_env_vars = PHP_

; Restrict PHP information leakage
expose_php = Off

; Log all errors
log_errors = On

; Do not register globals for input data
register_globals = Off

; Minimize allowable PHP post size
post_max_size = 1K

; Ensure PHP redirects appropriately
cgi.force_redirect = 0

; Disallow uploading unless necessary
file_uploads = Off

; Enable SQL safe mode
sql.safe_mode = On
mysqli.default_host = "192.168.1.252"
mysqli.default_port = "3306"
; Don't forget to change the user name and password
mysqli.default_user = "userNameHere"
mysqli.default_pw = "PasswordHere"
; Avoid Opening remote files
allow_url_fopen = Off

Restart PHP and nginx:

sudo systemctl restart php8.x-fpm.service
sudo systemctl restart nginx.service

Bonus

I've noticed that the background image is taking whopping 13.46 seconds to load and weights 1.3MB. Compressed it down to 455kB and reduced loading time down to 1.45s. Changed the file format from jpeg to webp and reduced size down to 130kB and loading time to 0.9s.

Install nginx brotli modules:

sudo apt update
sudo apt install libnginx-mod-http-brotli-filter libnginx-mod-http-brotli-static
ls /usr/lib/nginx/modules/ | grep brotli

Enable brotli (if you have include /etc/nginx/modules-enabled/*.conf; ignore it):

sudo nano /etc/nginx/nginx.conf
load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so;

http {
gzip on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_proxied any;
gzip_vary on;
gzip_types
application/javascript
application/json
application/xml
text/css
text/html
text/javascript
text/plain
text/xml
image/svg+xml;

brotli on;
brotli_static on;
brotli_comp_level 6;
brotli_min_length 256;
brotli_buffers 16 8k;
brotli_window 512k;
brotli_types
application/javascript
application/json
application/manifest+json
application/xml
application/x-font-ttf
font/opentype
image/svg+xml
image/x-icon
text/css
text/html
text/javascript
text/plain
text/xml;
}

Restart nginx:

sudo systemctl restart nginx

Bonus 2 Minify js

Just print js paths:

find . -type f -name '*.js' ! -name '*.min.js' -print0 | xargs -0 -n1 printf '%s\n'

Minify all .js files inside current dir and save as *.min.js:

find . -type f -name "*.js" ! -name "*.min.js" -print0   | xargs -0 -n1 -P4 -I{} sh -c 'curl -X POST -s --data-urlencode "input@${1}" https://www.toptal.com/developers/javascript-minifier/api/raw > "${1%.js}.min.js"' _ {}

Swap *.js with *.min.js:

swap-js-min.sh

#!/usr/bin/env bash
set -euo pipefail

# Swap .js and .min.js for files under the current directory.
# - If both exist: swap names atomically using a temp suffix.
# - If only one exists: rename it to the other extension.

shopt -s nullglob

tmp_suffix=".$(id -u)-$$.swap"

declare -A seen

# Collect unique basenames (strip trailing .min and .js)
while IFS= read -r -d '' f; do
  # remove trailing .js
  base="${f%.js}"
  # if it was foo.min.js this leaves foo.min -> strip .min
  base="${base%.min}"
  seen["$base"]=1
done < <(find . -type f ( -name "*.js" -o -name "*.min.js" ) -print0)

for base in "${!seen[@]}"; do
  js="$base.js"
  min="$base.min.js"

  if [[ -e "$js" && -e "$min" ]]; then
    mv -- "$js" "$js${tmp_suffix}"
    mv -- "$min" "$js"
    mv -- "$js${tmp_suffix}" "$min"
    printf 'swapped: %s <-> %sn' "$js" "$min"
  elif [[ -e "$js" ]]; then
    mv -- "$js" "$min"
    printf 'renamed: %s -> %sn' "$js" "$min"
  elif [[ -e "$min" ]]; then
    mv -- "$min" "$js"
    printf 'renamed: %s -> %sn' "$min" "$js"
  fi
done
chmod +x swap-js-min.sh
./swap-js-min.sh
rm ./swap-js-min.sh

Recommendations

Thanks for the reading. I hope you've enjoyed the articles alongside a cup of tea ☕