Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.

Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.

Rolled back to the backup before I made it public and now I have a security checklist.

  • Fair Fairy@thelemmy.club
    link
    fedilink
    arrow-up
    12
    ·
    1 hour ago

    I’m confused. I never disable root user and never got hacked.

    Is the issue that the app is coded in a shitty way maybe ?

    • Xanza@lemm.ee
      link
      fedilink
      English
      arrow-up
      2
      ·
      10 minutes ago

      You can’t really disable the root user. You can make it so they can’t login remotely, which is highly suggested.

    • cley_faye@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      47 minutes ago

      You can’t really disable it anyway.

      Hardening is mostly prevent root login from outside in case every other layer of authentication and access control broke, do not allow regular user to su/sudo into it for free, and have a tight grip on anything that’s executable and have a setuid bit set. I did not install a system from scratch in a long time but I believe this would be the default on most things that are not geared toward end-user devices, too.

  • MonkderVierte@lemmy.ml
    link
    fedilink
    arrow-up
    3
    ·
    2 hours ago

    Yeah, about this; any ssh server that can be run as user and doesn’t do shenanigans like switching user?

  • recklessengagement@lemmy.world
    link
    fedilink
    arrow-up
    22
    ·
    8 hours ago

    This sounds like something everyone should go through at least once, to underscore the importance of hardening that can be easily taken for granted

  • ohshit604@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    16
    ·
    edit-2
    8 hours ago

    I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.

    • Valmond@lemmy.world
      link
      fedilink
      arrow-up
      2
      ·
      4 hours ago

      I’m interested, how do you expose your services (on your PC I assume) to the internet through wireguard? Is it theough some VPN?

      • ohshit604@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        1
        ·
        2 hours ago

        VPN’s are neat, besides the fact they’re capable of masking your IP & DNS they’re also capable of providing resources to devices outside a network.

        A good example is the server at my work is only accessible on my works network, to access the server remotely without exposing it directly to the internet would be to use a VPN tunnel.

      • Zanathos@lemmy.world
        link
        fedilink
        arrow-up
        3
        ·
        2 hours ago

        Wireguard IS a VPN. He has somehow through his challenges of exposing services to the internet, exposed wireguard from his home to the internet for him to connect to. Then he can connect to his internal services from there.

        It’s honestly the best option and how I operate as well. I only have a handful of items exposed and even those flow through a DMZ proxy before hitting their destination servers.

  • nonentity@sh.itjust.works
    link
    fedilink
    arrow-up
    19
    ·
    9 hours ago

    Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.

    The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.

  • DaCrazyJamez@sh.itjust.works
    link
    fedilink
    arrow-up
    29
    ·
    11 hours ago

    As a linux n00b who just recently took the plunge and set up a public site (tho really just for my own / selfhosting),

    Can anyone recommend a good guide or starting place for how to harden the setup? Im running mint on my former gaming rig, site is set up LAMP

  • Hozerkiller@lemmy.ca
    link
    fedilink
    English
    arrow-up
    37
    ·
    13 hours ago

    I’ve gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn’t know better, instead of pure salt like another site I won’t mention.

    • Tablaste@linux.communityOP
      link
      fedilink
      English
      arrow-up
      24
      ·
      13 hours ago

      I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.

      We all make mistakes!

      • LordCrom@lemmy.world
        link
        fedilink
        arrow-up
        11
        ·
        11 hours ago

        There’s a 40 year I.T. veteran here that still suffers imposter syndrome. It’s a real thing I’ve never been able to shake off

        • pulsewidth@lemmy.world
          link
          fedilink
          arrow-up
          6
          ·
          9 hours ago

          Just look at who is in the White House, mate - and not just the president, but basically you can pick anyone he’s hand-picked for his staff.

          Surely that’s an instant cure for any qualified person feeling imposter syndrome in their job.

  • phx@lemmy.ca
    link
    fedilink
    arrow-up
    19
    ·
    12 hours ago

    Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.

    When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.

    • DefederateLemmyMl@feddit.nl
      link
      fedilink
      arrow-up
      1
      ·
      1 hour ago

      Do not allow username/password login for ssh

      This is disabled by default for the root user.

      $ man sshd_config
      
      ...
             PermitRootLogin
                     Specifies whether root  can  log  in  using  ssh(1).   The  argument  must  be  yes,  prohibit-password,
                     forced-commands-only, or no.  The default is prohibit-password.
      ...
      
      
    • LordCrom@lemmy.world
      link
      fedilink
      arrow-up
      9
      ·
      11 hours ago

      If it’s public facing, how about dont turn on ssh to the public, open it to select ips or ranges. Use a non standard port, use a cert or even a radius with TOTP like privacyIdea. How about a port knocker to open the non standard port as well. Autoban to lock out source ips.

      That’s just off the top of my head.

      There’s a lot you can do to harden a host.

  • communism@lemmy.ml
    link
    fedilink
    arrow-up
    37
    ·
    17 hours ago

    How are people’s servers getting compromised? I’m no security expert (I’ve never worked in tech at all) and have a public VPS, never been compromised. Mainly just use SSH keys not passwords, I don’t do anything too crazy. Like if you have open SSH on port 22 with root login enabled and your root password is password123 then maybe but I’m surprised I’ve never been pwned if it’s so easy to get got…

    • cmnybo@discuss.tchncs.de
      link
      fedilink
      English
      arrow-up
      26
      ·
      16 hours ago

      By allowing password login and using weak passwords or by reusing passwords that have been involved in a data breach somewhere.

      • communism@lemmy.ml
        link
        fedilink
        arrow-up
        7
        ·
        16 hours ago

        That makes sense. It feels a bit mad that the difference between getting pwned super easy vs not is something simple like that. But also reassuring to know, cause I was wondering how I heard about so many hobbyist home labs etc getting compromised when it’d be pretty hard to obtain a reasonably secured private key (ie not uploaded onto the cloud or anything, not stored on an unencrypted drive that other people can easily access, etc). But if it’s just password logins that makes more sense.

    • pageflight@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      ·
      16 hours ago

      The one db I saw compromised at a previous employer was an AWS RDS with public Internet access open and default admin username/password. Luckily it was just full of test data, so when we noticed its contents had been replaced with a ransom message we just deleted the instance.

  • Punkie@lemmy.world
    link
    fedilink
    arrow-up
    151
    ·
    21 hours ago

    Basic setup for me is scripted on a new system. In regards to ssh, I make sure:

    • Root account is disabled, sudo only
    • ssh only by keys
    • sshd blocks all users but a few, via AllowUsers
    • All ‘default usernames’ are removed, like ec2-user or ubuntu for AWS ec2 systems
    • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.
    • Services are only allowed connections by an allow list of IPs or subnets. Internal, when possible.

    My systems are not “unhackable” but not low-hanging fruit, either. I assume everything I have out there can be hacked by someone SUPER determined, and have a vector of protection to mitigate backwash in case they gain full access.

    • feddylemmy@lemmy.world
      link
      fedilink
      arrow-up
      66
      ·
      19 hours ago
      • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.

      Gosh I get unreasonably frustrated when someone says yeah but that’s just security through obscurity. Like yeah, we all know what nmap is, a persistent threat will just look at all 65535 and figure out where ssh is listening… But if you change your threat model and talk about bots? Logs are much cleaner and moving ports gets rid of a lot of traffic. Obviously so does enabling keys only.

      Also does anyone still port knock these days?

      • josefo@leminal.space
        link
        fedilink
        arrow-up
        4
        ·
        11 hours ago

        Literally the only time I got somewhat hacked was when I left the default port of the service. Obscurity is reasonable, combined with other things like the ones mentioned here make you pretty much invulnerable to casuals. Somebody needs to target you to get anything.

      • kernelle@0d.gs
        link
        fedilink
        arrow-up
        17
        ·
        18 hours ago

        Also does anyone still port knock these days?

        Enter Masscan, probably a net negative for the internet, so use with care.

        • davidgro@lemmy.world
          link
          fedilink
          arrow-up
          6
          ·
          17 hours ago

          I didn’t see anything about port knocking there, it rather looks like it has the opposite focus - a quote from that page is “features that support widespread scanning of many machines are supported, while in-depth scanning of single machines aren’t.”

          • kernelle@0d.gs
            link
            fedilink
            arrow-up
            2
            ·
            16 hours ago

            Sure yeah it’s a discovery tool OOTB, but I’ve used it to perform specific packet sequences as well.

  • kibiz0r@midwest.social
    link
    fedilink
    English
    arrow-up
    70
    ·
    20 hours ago

    One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.

    “Hey, why is this Valheim server running like shit?”

    “Wtf is xrx?”

    “Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”

    So anyway, now I use NixOS.

    • pageflight@lemmy.world
      link
      fedilink
      English
      arrow-up
      14
      ·
      16 hours ago

      Good point about a default deny approach to users and ssh, so random services don’t add insecure logins.

  • mlg@lemmy.world
    link
    fedilink
    English
    arrow-up
    46
    ·
    19 hours ago

    Lol you can actually demo a github compromise in real time to an audience.

    Make a repo with an API key, publish it, and literally just watch as it takes only a few minutes before a script logs in.