Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.
Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.
Rolled back to the backup before I made it public and now I have a security checklist.
This sounds like something everyone should go through at least once, to underscore the importance of hardening that can be easily taken for granted
i use su -
Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.
The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.
I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.
As a linux n00b who just recently took the plunge and set up a public site (tho really just for my own / selfhosting),
Can anyone recommend a good guide or starting place for how to harden the setup? Im running mint on my former gaming rig, site is set up LAMP
The other poster gave you a lot. If that’s too much at once, the really low hanging fruit you want to start with is:
-
Choose an active, secure distro. There’s a lot of flavors of Linux out there and they can be fun to try but if you’re putting something up publicly it should be running on one that’s well maintained and known for security. CentOS and Debian are excellent easy choices for example.
-
Similarly, pick well maintained software with a track record. Nginx and Apache have been around forever and have excellent track records, for example, both for being secure and fixing flaws quickly.
-
If you use Docker, once again keep an eye out for things that are actively maintained. If you decide to use Nginx, there will be five million containers to choose from. DockerHub gives you the tools to make this determination: Download number is a decent proxy for “how many people are using this” and the list of updates tells you how often and how recently it’s being updated.
-
Finally, definitely do look at the other poster’s notes about SSH. 5 seconds after you put up an SSH server, you’ll be getting hit with rogue login attempts.
-
Definitely get a password manager, and it’s not just one password per server but one password per service. Your login password to the computer is different from your login to any other things your server is running.
The rest requires research, but these steps will protect you from the most common threats pretty effectively. The world is full of bots poking at every service they can find, so keeping them out is crucial. You won’t be protected from a dedicated, knowledgeable attacker until you do the rest of what the other poster said, and then some, so try not to make too many enemies.
The TLDR is here : https://www.digitalocean.com/community/tutorials/recommended-security-measures-to-protect-your-servers
You won’t be protected from a dedicated, knowledgeable attacker until you do the rest of what the other poster said, and then some,
You’re right I didn’t even get to ACME and PKI or TOTP
https://letsencrypt.org/getting-started/
https://openbao.org/docs/secrets/pki/
https://openbao.org/docs/secrets/totp/
And for bonus points build your own certificate authority to sign it all.
https://smallstep.com/blog/build-a-tiny-ca-with-raspberry-pi-yubikey/
-
Paranoid external security. I’m assuming you already have a domain name. I’m also assuming you have some ICANN anonymization setup.
This is your local reverse Proxy. You can manage all this with a container called nginx proxy manager, but it could benefit you to know it’s inner workings first. https://www.howtogeek.com/devops/what-is-a-reverse-proxy-and-how-does-it-work/
https://cloud9sc.com/nginx-proxy-manager-hardening/
https://github.com/NginxProxyManager/nginx-proxy-manager
Next you’ll want to proxy your IP address as you don’t want that pointing to your home address
https://developers.cloudflare.com/learning-paths/get-started-free/onboarding/proxy-dns-records/
Remote access is next. I would suggest setting up wireguard on a machine that’s not your webserver, but you can also set that up in a container as well. Either way you’ll need to punch another hole in your router to point to your wire guard bastion host on your local network. It has many clients for windows and linux and android and IOS
https://github.com/angristan/wireguard-install
https://www.wireguard.com/quickstart/
https://github.com/linuxserver/docker-wireguard
Now internally, I’m assuming you’re using Linux. In that case I’d suggest securing your ssh on all machines that you log into. On the machines you’re running you should also install fail2ban, UFW, git, and some monitoring if you have the overhead but the monitoring part is outside of the purview of this comment. If you’re using UFW your very first command should be
sudo ufw allow sshhttps://www.howtogeek.com/443156/the-best-ways-to-secure-your-ssh-server/
https://github.com/fail2ban/fail2ban
https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands
Now for securing internal linux harden the kernel and remove root user. If you do this you should have a password manager setup. keepassx or bitwarden are ones I like. If those suck I’m sure someone will suggest something better. The password manager will have the root password for all of your Linux machines and they should be different passwords.
https://www.makeuseof.com/ways-improve-linux-user-account-security/
https://bitwarden.com/help/self-host-an-organization/
Finally you can harden the kernel
https://codezup.com/the-ultimate-guide-to-hardening-your-linux-system-with-kernel-parameters/
TLDR: it takes research but a good place to start is here
Correct, horse battery staple.
I couldn’t justify putting correct in my username on Lemmy. But I loved the reference too much not to use it, so here I am, a less secure truncated version of a better password.
I’ve gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn’t know better, instead of pure salt like another site I won’t mention.
I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.
We all make mistakes!
There’s a 40 year I.T. veteran here that still suffers imposter syndrome. It’s a real thing I’ve never been able to shake off
Just look at who is in the White House, mate - and not just the president, but basically you can pick anyone he’s hand-picked for his staff.
Surely that’s an instant cure for any qualified person feeling imposter syndrome in their job.
Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.
When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.
questionable?
Yeah just like that. Ask more questions
Do not allow username/password login for ssh. Force certificate authentication only!
If it’s public facing, how about dont turn on ssh to the public, open it to select ips or ranges. Use a non standard port, use a cert or even a radius with TOTP like privacyIdea. How about a port knocker to open the non standard port as well. Autoban to lock out source ips.
That’s just off the top of my head.
There’s a lot you can do to harden a host.
I like to spin up a public facing server and run tcpdump
Lol! Honeypot or just bored?
Actually I was troubleshooting a Firewall issue on site. I just forgot to use the filter arguments to reduce the output.
Oooof, I know that feeling.
How are people’s servers getting compromised? I’m no security expert (I’ve never worked in tech at all) and have a public VPS, never been compromised. Mainly just use SSH keys not passwords, I don’t do anything too crazy. Like if you have open SSH on port 22 with root login enabled and your root password is
password123then maybe but I’m surprised I’ve never been pwned if it’s so easy to get got…glad my root pass is
toorand not something as obvious aspassword123toor, like Tor, the leet hacker software. So it must be super secure.
By allowing password login and using weak passwords or by reusing passwords that have been involved in a data breach somewhere.
That makes sense. It feels a bit mad that the difference between getting pwned super easy vs not is something simple like that. But also reassuring to know, cause I was wondering how I heard about so many hobbyist home labs etc getting compromised when it’d be pretty hard to obtain a reasonably secured private key (ie not uploaded onto the cloud or anything, not stored on an unencrypted drive that other people can easily access, etc). But if it’s just password logins that makes more sense.
That’s incredible, I’ve got the same combination on my luggage.
The one db I saw compromised at a previous employer was an AWS RDS with public Internet access open and default admin username/password. Luckily it was just full of test data, so when we noticed its contents had been replaced with a ransom message we just deleted the instance.
Use gnome powder to shrink, go behind the counter, kick his ass and get your money back.
Basic setup for me is scripted on a new system. In regards to ssh, I make sure:
- Root account is disabled, sudo only
- ssh only by keys
- sshd blocks all users but a few, via AllowUsers
- All ‘default usernames’ are removed, like ec2-user or ubuntu for AWS ec2 systems
- The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.
- Services are only allowed connections by an allow list of IPs or subnets. Internal, when possible.
My systems are not “unhackable” but not low-hanging fruit, either. I assume everything I have out there can be hacked by someone SUPER determined, and have a vector of protection to mitigate backwash in case they gain full access.
- The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.
Gosh I get unreasonably frustrated when someone says yeah but that’s just security through obscurity. Like yeah, we all know what nmap is, a persistent threat will just look at all 65535 and figure out where ssh is listening… But if you change your threat model and talk about bots? Logs are much cleaner and moving ports gets rid of a lot of traffic. Obviously so does enabling keys only.
Also does anyone still port knock these days?
I use port knock. Really helps against scans if you are the edge device.
Literally the only time I got somewhat hacked was when I left the default port of the service. Obscurity is reasonable, combined with other things like the ones mentioned here make you pretty much invulnerable to casuals. Somebody needs to target you to get anything.
Also does anyone still port knock these days?
If they did, would we know?
Also does anyone still port knock these days?
Enter Masscan, probably a net negative for the internet, so use with care.
I didn’t see anything about port knocking there, it rather looks like it has the opposite focus - a quote from that page is “features that support widespread scanning of many machines are supported, while in-depth scanning of single machines aren’t.”
Sure yeah it’s a discovery tool OOTB, but I’ve used it to perform specific packet sequences as well.
Lol you can actually demo a github compromise in real time to an audience.
Make a repo with an API key, publish it, and literally just watch as it takes only a few minutes before a script logs in.
I search commits for “removed env file” to hopefully catch people who don’t know how git works.
–verbose please?
edit: never mind, found it. So there’s dumbasses storing sensitive data (keys!) inside their git folder and unable to configure .gitignore…
My work is transferring to github from svn currently
My condolences
yeah, I just tried it there, people actually did it.
You gremlin lmao
One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.
“Hey, why is this Valheim server running like shit?”
“Wtf is
xrx?”“Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”
So anyway, now I use NixOS.
Good point about a default deny approach to users and ssh, so random services don’t add insecure logins.
Interesting. Do you know how it got compromised?
I published it to the internet and the next day, I couldn’t ssh into the server anymore with my user account and something was off.
Tried root + password, also failed.
Immediately facepalmed because the password was the generic 8 characters and there was no fail2ban to stop guessing.
Don’t use passwords for ssh. Use keys and disable password authentication.
More importantly, don’t open up SSH to public access. Use a VPN connection to the server. This is really easy to do with Netbird, Tailscale, etc. You should only ever be able to connect to SSH privately, never over the public net.
Tailscale? Netbird? I have been using hamachi like a fucking neanderthal. I love this posts, I learn so much
what’s netbird
https://netbird.io/. Wireguard based software defined networking, very similar to Tailscale.
It’s perfectly safe to run SSH on port 22 towards the open Internet with public key authentication only.
https://nvd.nist.gov/vuln/detail/cve-2024-6409 RCE as root without authentication via Open SSH. If they’ve got a connection, that’s more than nothing and sometimes it’s enough.
That attack vector is exactly the same towards a VPN.
wow crazy that this was the default setup. It should really force you to either disable root or set a proper password (or warn you)
Most distributions disable root by default
Which ones? I’m asking because that isn’t true for cent, rocky, arch.
Mostly Ubuntu. And… I think it’s just Ubuntu.
Fedora (immutable at least) has it disabled by default I think, but it’s just one checkbox away in one of the setup menus.
Standard Fedora does as well
we’re probably talking about different things. virtually no distribution comes with root access with a password. you have to explicitly give the root user a password. without a password no amount of brute force sshing root will work. I’m not saying the root user is entirely disabled. so either the service OP is building on is basically a goldmine for compromised machines or OP literally shot themselves in the root by giving root a password manually. something you should never do.
Many cloud providers (the cheap ones in particular) will put patches on top of the base distro, so sometimes root always gets a password. Even for Ubuntu.
There are ways around this, like proper cloud-init support, but not exactly beginner friendly.
#no thank you lol
Rocky asks during setup, I assume centOS too
Love Hetzner. You just give them your public key and they boot you into a rescue system from which you can install what you want how you want.
I think their auction servers are a hidden gem. I mean the prices used to be better. Now they have some kind of systrem that resets them when they get too low. But the prices are still pretty good I think. But a year or two ago I got a pretty good deal on two decently spec’d servers.
People are scared off by the fact you just get their rescue prompt on auctions boxes… Except their rescue prompt has a guided imaging setup tool to install pretty much every popular distro with configurable raid options etc.
Yeah, I basically jump from auction system to auction system every other year or so and either get a cheaper or more powerful server or both.
I monitor for good deals. Because there’s no contract it’s easy to add one, move stuff over at your leisure and kill the old one off. It’s the better way to do it for semi serious stuff.
Now that you mentioned it, it didn’t! I recall even docker Linux setups would yell at me.
because the password was the generic 8 characters and there was no fail2ban to stop guessing
Oof yea that’ll do it, your usually fine as long as you hardened enough to at least ward off the script kiddies. The people with actual real skill tend to go after…juicer targets lmao
Haha I’m pretty sure my little server was just part of the “let’s test our dumb script to see if it works. Oh wow it did what a moron!”
Lessons learned.
Lol ssh has no reason to be port exposed in 99% of home server setups.
VPNs are extremely easy, free, and wireguard is very performant with openvpn also fine for ssh. I have yet to see any usecase for simply port forwarding ssh in a home setup. Even a public git server can be tunneled through https.
Yeah I’m honest with myself that I’m a security newb and don’t know how to even know what I’m vulnerable to yet. So I didn’t bother opening anything at all on my router. That sounded way too scary.
Tailscale really is magic. I just use Cloudflare to forward a domain I own, and I can get to my services, my NextCloud, everything, from anywhere, and I’m reasonably confident I’m not exposing any doors to the innumerable botnet swarms.
It might be a tiny bit inconvenient if I wanted to serve anything to anyone not in my Tailnet or already on my home LAN (like sending al someone a link to a NextCloud folder for instance.), but at this point, that’s quite the edge case.
I learned to set up NGINX proxy manager for a reverse proxy though, and that’s pretty great! I still harden stuff where I can as I learn, even though I’m confident nobody’s even seeing it.
I ran a standard raspian ssh server on my home network for several years, default user was removed and my own user was in it’s place, root was configured as standard on a raspbian, my account had a complex but fairly short password, no specific keys set.
I saw constant attacks but to my knowledge, it was never breached.
I removed it when I realized that my ISP might take a dim view of running a server on their home client net that they didn’t know about, especially since it showed up on Shodan…
Don’t do what I did, secure your systems properly!
But it was kinda cool to be able to SSH from Thailand back home to Sweden and browse my NAS, it was super slow, but damn cool…
But it was kinda cool to be able to SSH from Thailand back home to Sweden and browse my NAS, it was super slow, but damn cool…
That feels like sorcery, doesn’t it? You can still do this WAY safer by using Wirehuard or Tailscale. I use Tailscale myself to VPN to my NAS.
I get a kick out of showing people my NextCloud Memories albums or Jellyfin videos from my phone and saying “This is talking to the box in my house right now! Isn’t that cool!?” Hahaha.
Why would a Swedish ISP care? I’ve run servers from home since I first connected up in … 1996. I’ve had a lot of different ISPs during that time, although nowadays I always choose Bahnhof because of them fighting the good fights.
They probably don’t, unless I got compromised and bad traffic came from their network, but I was paranoid, and wanted to avoid the possibility.
Which distro allows root to login via SSH?
All of them if you configure it?
Not very many. None of the enterprise ones, at least.
Ah, timeless classic.
Any idea what ip addresses were used to compromise it?
