custom.d

Bad Bot Blocker

An amusing of web traffic are automated bots either trying to send referral spam, looking for vulnerabilities, and other nonsense!

This started off as a fork of a popular bot blocker and has morphed into a general firewall of sorts for my servers. I double check and add IPs and referrers based on my server logs.

• Apache upstream - apache-ultimate-bad-bot-blocker
  
• Nginx upstream - nginx-ultimate-bad-bot-blocker
• Sync bad-referrer-words.conf - matomo-org/referrer-spam-blacklists
  
• Sync with existing whitelist-ips.conf & blacklist-ips.conf
  
• Check other IPs reported @ https://www.abuseipdb.com
• IP/DNS references @ awesome-threat-intelligence

Init

custom.d goes in apache & conf.d goes in nginx

#add to nginx/apache.conf

sudo vi /etc/apache2/apache.conf
<Location "/">
  AuthMerging And
  Include custom.d/globalblacklist.conf
</Location>
sudo systemctl reload apache2

sudo vi /etc/nginx/nginx.conf
include /etc/nginx/conf.d/*;

sudo vi /etc/nginx/sites-available/default
server {
  include /etc/nginx/bots.d/blockbots.conf;
  include /etc/nginx/bots.d/ddos.conf;
}
sudo systemctl reload nginx

Sync remote host

cd /etc/apache2/   
git clone https://github.com/windhamdavid/custom.d/   
cd custom.d  
sudo git pull origin/code master
sudo apache2ctl configtest
sudo service apache2 reload

keep it sync'd with upstream

edit/prune upstream on localhost

git checkout master
git fetch upstream   
git merge upstream/master  
(edit README.md, git add, git commit)
git merge upstream/master
git filter-branch -f --prune-empty --subdirectory-filter Apache_2.4/custom.d master   
gpom #git push origin master   
gpcm #git push code master   

---

Notes

23.06.12

• new IPs added from logs on Zeke and Woozie

23.03.05

• new IPs added from logs on Zeke and Woozie

re: sync referrer-words:

• always forget to tap ⌥ to get multiple row carets. ⌘ → to end of line.

23.02.11

• whitelisted a new server and watched the logs to block out some bots and other domains that were already hitting the IP before got the domains rolling.

2021/03

• current branch was behind remote. Forgot I had whitelisted Screaming Frog in a previous commit on the remote host. Used -f to overwrite.

2022/02

• updated to Version: V3.2022.02.1316
• sync'd referrers and added some custom referrers and IPs.
• rm screaming 🐸 from globalblacklist so I can use it.
• since the IP blacklist is not really kept up to date, I'm using IPs gathered from several list @ https://github.com/hslatman/awesome-threat-intelligence

2022/06

• updated to Version: V3.2022.05.1398
• added IPs and referrers from server logs
• easy to check IP abuse @ https://www.abuseipdb.com
• added sudo command to git pull so that files retain root permissions. see: https://github.blog/2022-04-12-git-security-vulnerability-announced/