[HN Gopher] Show HN: A proactive way of securing your Linux server
___________________________________________________________________
Show HN: A proactive way of securing your Linux server
Author : prashantgupta24
Score : 79 points
Date : 2020-09-03 16:38 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| fl0wenol wrote:
| The idea is better than port knocking, in the sense that you take
| active action to associate your host with the server, but there's
| a few issues.
|
| * Something this simple shouldn't need k8s, unless it was
| intended as an exercise by the developer for that reason
|
| * It combines the idea of using a non-standard port with
| certificate based authentication, which you can already do with
| SSH-- it's functionally the same with more steps
|
| Hypothetically, this approach can be more powerful as a
| centralized service running elsewhere, (i.e. cloud, your remote
| DC) and used for a whole bunch of jump boxes and end-users. End-
| users could run a "check-in" script wrapping around SSH that
| notified the service that a user was imminent, and then could
| check the server to see if 1) The bearer token is accepted for
| the destination server 2) That the destination server checked in,
| saw the new incoming request, and has already opened the port --
| And then proceeds to run the SSH command if all is well, or fails
| with appropriate error.
| Tepix wrote:
| Running SSH on a non-standard port gets rid of the brute force
| login attempts, too.
| zootboy wrote:
| In my experience, a non-standard port reduces the number of
| brute force attempts, but it doesn't eliminate them. I just do
| a simple rate limit on syn packets to the port and use SSH keys
| for auth, and that feels like plenty of security.
| john-shaffer wrote:
| On EC2, I've had a great experience using AWS Systems Manager
| [1]. I don't need any ports open, and it works great with normal
| shell tools and emacs given a .ssh/config like this:
|
| host i-* mi-*
|
| ProxyCommand sh -c "aws ssm start-session --target %h --document-
| name AWS-StartSSHSession --parameters 'portNumber=%p'"
|
| [1] https://docs.aws.amazon.com/systems-
| manager/latest/userguide...
| swinglock wrote:
| A proactive way of increasing your attack surface no doubt.
| joana035 wrote:
| A proactive way to shoot yourself in the foot for sure.
| oehtXRwMkIs wrote:
| If I have public key authentication set up for ssh, should I even
| bother with fail2ban, firewalld-rest, port knocking, etc.?
| There's no way anyone is brute forcing my ed25519 key, so what's
| the point? Sure security should be layered and all that but it
| seems like public key auth is so strong by itself anything on top
| seems unnecessary.
| jooize wrote:
| Might protect against some ludicrous attack that we're not
| considering. Limiting attempts seems to me to always be a good
| idea.
| RL_Quine wrote:
| There's no point to this at all, it's just making services more
| fragile.
|
| If "brute force" attacks are a concern, you are a fool that has
| password authentication turned on.
|
| If you have password authentication turned off, why does it
| matter?
| yearoflinux wrote:
| You are reducing attack vectors. How do you know your ssh
| implementation is secure?
| avtar wrote:
| > If I have public key authentication set up for ssh, should I
| even bother with fail2ban
|
| If someone's bruteforcing their way in via ssh it's probably
| worthwhile blocking them before they start trying other
| services.
| fmajid wrote:
| It's very easy for brute-force attacks from botnets to consume
| all your SSH process slots and then you are locked out.
| roveo wrote:
| Can confirm, had this problem the other day. CI started
| failing and after a short investigation I figured out sshd
| was just dropping my connections. Installed fail2ban first,
| but then just used a firewall since all valid IPs are known.
| joana035 wrote:
| That ain't the truth.
| koolba wrote:
| Adding a JWT authenticated API layer to something is not a first
| choice for adding additional security.
|
| If you want something like this look into platform level
| firewalls (ex: AWS security groups) or run spiped in front of
| your SSH server. I'd trust that a hell of a lot more than a REST
| API.
| yabones wrote:
| I don't like this approach very much because it's much more
| complicated than it would have to be. Security is about layers,
| and this is essentially one layer that acts as the sole guardian
| for sshd.
|
| The way that I like to do this is to have a common 'entry point'
| for all my cloud systems. Instead of whitelisting IPs on every
| VPS or cluster I build, I just add them to the ACL on my
| management server. All the other systems only allow SSH
| connections in from the bastion server. In practice, it works
| like this:
|
| * Add IP to the whitelist file in my change control
|
| * Run the Terraform script to update the DigitalOcean ACL
|
| * Start an SSH agent locally and add the key for the bastion, as
| well as the key for the destination
|
| * Connect to the destination server by using ProxyJump
|
| So, connecting to a box would always route through my bastion
| system first, like this: ssh -J
| mgmt.mydomain.net cool-app-server.mydomain.net
|
| I've been doing this for a couple years, and it works great. I
| practically never have attempted login attempts on my systems.
| And, since I use an ssh agent to forward the keys through the
| bastion without ever actually storing them, a compromise of that
| system doesn't really give the attacker anything other than
| access to port 22 on a bunch of systems that they wouldn't know
| where to find. Only the most sophisticated attack
| (https://xkcd.com/538/) would lead to a real compromise.
| jiggawatts wrote:
| I love that XKCD, but this particular cartoon gives a false
| impression in order to be humorous. (You can't fault a comic
| artist for this of course.)
|
| In my experience the #1 _real_ attack vector is the shouty
| middle manager demanding that: "Everything be opened right now
| because some outsourced admin says he needed it".
|
| It's nigh impossible to use technological measures to defeat a
| CxO or a department head.
| gregod wrote:
| If you do not need the more granular firewall configuration
| options there is also classic port knocking
| (https://en.wikipedia.org/wiki/Port_knocking) where the daemon
| sits behind the firewall so all ports can be closed by default.
| ff7c11 wrote:
| Have you tried knockd https://linux.die.net/man/1/knockd ? You
| send a special sequence of "knocks" to the server (packets to
| different ports) and it executes a command such as allowing your
| IP for a time period. No JWTs.
| yearoflinux wrote:
| This is very nice and renders the OP useless. OP itself is
| another attack vector. Do you know what syscall API knockd uses
| to listen to "link" ?
| geocrasher wrote:
| This is interesting in that it combines the concept of port
| knocking with a REST interface, which I'm assuming is up to the
| user to create a front end for.
|
| Unfotunately it also relies on Kubernetes which means that using
| it for a single system isn't practical. At least, not for _this_
| server owner.
|
| My own approach is simply security by obscurity (a non standard
| port) with APF/BFD doing the needful for locking bots out if they
| figure out the port. I've had to change ports only once in 6
| years, so it's working to keep bots out rather nicely.
|
| And really that's all these things are- a way to keep bots out. A
| determined attacker will figure this stuff out anyway.
| CamJN wrote:
| 6. Possible enhancements
|
| Rate limiting the number of requests that can be made to the
| application
|
| So this just moves the brute forcing target from ssh to a web
| app. A lot of work for no added security.
| kelnos wrote:
| This just feels like moving ssh auth to another port, using an
| obscure "authentication format" that nearly no one uses.
|
| If you're using pubkey or certificate auth, have disabled
| password auth, and restrict what users are allowed to ssh in,
| this just feels like added complexity and points of failure (not
| to mention a possible source of crypto vulnerabilities) with not
| much added benefit.
|
| Having said that, it's a cool project in general, and I feel like
| it could be useful to dynamically manage access to large groups
| of servers (perhaps not just ssh; you could use it to manage
| access to https interfaces or other things). Then again, if you
| have a large group of servers, you should have those ports
| blocked to the world and only allow access through a VPN and/or
| jump boxes.
| codetrotter wrote:
| > Then again, if you have a large group of servers, you should
| have those ports blocked to the world and only allow access
| through a VPN and/or jump boxes.
|
| Even with just one server it can make sense to do this.
|
| In my case I am running Wireguard on my server, and I use that
| VPN among other things for being able to remote in to my
| grandfather's computer when he needs help. Since we are both
| behind NAT at our respective homes, a VPN was needed and I
| chose to use Wireguard. Wireguard is super easy to set up, and
| works on FreeBSD (my server and also my desktop run this),
| macOS (my laptop and my grandfather's desktop run this), iOS
| (my phone) and Linux (currently I don't run Linux but I have
| for many years and will certainly run it again on some machines
| again). I don't use Windows, almost haven't for a decade with a
| few small exceptions, but I think Wireguard is available for
| Windows also.
|
| So anyway, I decided back when I set up Wireguard hey, now that
| all my machines and my phone is on this VPN, there's really no
| reason to expose SSH to the internet anymore. So ever since
| then I have had sshd listen only on the VPN. It works great.
| jackric wrote:
| Agreed - it looks like security through obscurity to me. It has
| value by shifting the goalposts away from common mass scanning,
| but to a motivated attacker the security is equivalent.
| treve wrote:
| Relying on security through obscurity alone is a terrible
| idea, but there is absolutely a place for it. Most of us do
| not specifically get targeted, but will be affected by
| automated attacks that look at a large block of ips.
| daveidol wrote:
| Yeah, just moving your SSH daemon from port 22 to some
| other port number alone can help with that.
| deeblering4 wrote:
| Requiring ssh keys, disabling password auth, and using ssh
| bastion hosts is my preferred approach.
| robomc wrote:
| Yeah it's not clear to me why exposing the ability to configure
| your firewall, protected by a keypair, to the internet, is
| safer than just letting someone bang their head against sshd,
| protected by a keypair, and presumably having had a lot more
| eyes on it for security.
| donor20 wrote:
| And the distributions can also do the SElinux stuff that I
| don't use day to day.
| manexploitsman wrote:
| I like this "proactive" solution :) Endlessh: an SSH Tarpit
| https://nullprogram.com/blog/2019/03/22/
| nyx_ wrote:
| I like to run fail2ban in conjunction with a non-standard SSH
| port on which only public key auth is available.
|
| This way, most of the junkware that does rude things to port 22
| is banging on a closed door; the slightly more effective junkware
| that actually finds the SSH port gets banned immediately, because
| I know anyone trying to log in with a password is full of it.
| dmos62 wrote:
| Wouldn't it make more sense then to keep SSH on 22? Think of it
| like a honeypot. The more accessible you make it, the more bans
| you get. There was a guy on here saying that fail2ban banning
| SSH bruteforcers also reduced the number of HTTP bruteforcers,
| because they overlap.
| greenie_beans wrote:
| This is a noob question, but could you not setup fail2ban to
| ban bad IPs on both ports, even if you're not using SSH on
| one? I'm wondering if it's possible to close port 22 and set
| fail2ban rules to ban any IPs that try to SSH there, while
| changing your SSH port and also setting up fail2ban on that
| port, too. Does a closed port work like that?
| ses1984 wrote:
| You could have the firewall log connection attempts on port
| 22 and plug that into fail2ban.
| kelnos wrote:
| fail2ban works by tailing authentication logs and matching
| regexes to find failed attempts and the IP address of the
| source. So if nothing is listening on the port, there won't
| be anything to spit out logs that fail2ban can read.
|
| You could potentially create a fake daemon that listens on
| port 22 and just logs every access attempt, and set up
| fail2ban to block any IP that even opens a connection to
| the port, regardless of what they try to send over it.
|
| Actually, I think iptables has a LOG target that can log
| connection attempts even without something listening, so
| that could work too and be much simpler.
| mjoin wrote:
| > anyone trying to log in with a password is full of it Dang!
| But so true
| ed25519FUUU wrote:
| Secure SSH and you won't have to worry about rogue login
| attempts, which happen to everyone. If it really bothers you then
| move it to another port where it will happen less.
|
| But install a new firewall management system? Sounds like it will
| definitely introduce more risks than the problems it solves,
| which for the SSH example isn't really a problem at all.
| ryanjkirk wrote:
| Setting this up makes much less sense than setting up a tested
| vpn, such as wireguard or open, or even a persistent ssh tunnel
| using autossh to your home rpi.
|
| I would never allow my prod systems to be potentially exposed by
| an api that runs as root. (And the documentation is incorrect on
| that; it should run as an unprivileged user with sudo privs to
| only run a wrapper script that runs firewall-cmd).
|
| This also makes little sense in the context of configuration
| management, which should be enforcing a static set of iptables
| rules.
| stqism wrote:
| Personally I'd make a binary run as a non-root user with just
| CAP_NET_ADMIN and CAP_NET_RAW permissions to cut the scope down
| even more.
| ryanjkirk wrote:
| Having not used setcap, would that also require the setuid
| bit to be set with ownership as root (like sudo)? And reside
| on a filesystem without the nosuid mount option?
| wrkronmiller wrote:
| I usually just disable password auth and update regularly, when I
| have an SSH server open to the internet.
|
| Short of an 0day in the SSH service, I expect brute forcing the
| private key(s) to take longer than I have years to live.
| oceanghost wrote:
| There are bots out there that will attack from thousands if not
| 10's of thousands of IPs... It's impossible to ban the
| individual IPs byh hand, so I let them tell me who they are by
| leaving the SSH port open and have fail2ban ban any IP that
| fails to authenticate for a month.
|
| This had an unintended effect... most of my automated HTTP(s)
| attacks stopped as well-- presumably because the IPs of
| compromised machines were already blocked.
| tambourine_man wrote:
| But it takes CPU, fills up log files, etc.
|
| fail2ban is the first thing install. It's also easy to
| understand, which I love. It's basically regex on log files.
| RL_Quine wrote:
| What decade are your servers from? A failed key
| authentication doesn't even involve any cryptographic
| operation unless they also know the hash of your public key.
| gramakri wrote:
| There is no cpu use. The hash compare of key will fail if I
| understand correctly.
| necheffa wrote:
| It really doesn't make that big a difference. I have single
| core, 515MiB VMs that idle at <1% utilization with background
| noise. Most distros come with logrotate anyways.
| drdaeman wrote:
| > Most distros
|
| Most distros those days come with journald, and to best of
| my awareness journald does not have per-unit (or better,
| per unit per message type) configurable retention.
|
| Still in the TODO list today: https://github.com/systemd/sy
| stemd/blob/908dbc70d6abeb9f6562... (since at least 2016),
| with corresponding issue gathering no traction:
| https://github.com/systemd/systemd/issues/9519
| _jal wrote:
| The proper fix for journald is
|
| storage=none
|
| And continue logging as always. (Unless you use
| rsyslogd.)
| joana035 wrote:
| Nope nope nope.
|
| This has serious quality issues and any competent and nice
| sysadmin should say _nope_ to run this in any serious
| environment, for your own good ;-)
|
| IP addresses are just strings? At least parse/validate for
| IPv4/IPv6.
|
| Why yet another database? Can't the rules be loaded from the
| running system?
|
| Why not just ipset-persistent + knockd + portsentry? I know it is
| easy to get overexcited with a new pet project, but just be
| careful to not put this kind of stuff in a production system
| kidos.
| prashantgupta24 wrote:
| Thank you for the feedback!
| hitpointdrew wrote:
| Why is SSH open to the public internet at all, ever? Use
| WireGuard as VPN tunnel to your servers network, then SSH via
| internal IP.
| joana035 wrote:
| > Why is SSH open to the public internet at all, ever?
|
| Wasn't SSH made for that?
| kbenson wrote:
| Sometimes you literally just have a server on the internet, and
| there's not a network to connect to with wireguard. You could
| connect directly to the server with wireguard, but I actually
| trust OpenSSH more than VPN software built into the kernel (not
| to mention I can update it easily without rebooting).
| johnghanks wrote:
| Wow - why didn't I think of just setting up a firewall instead of
| running fail2ban. Great idea.
| _joel wrote:
| eh, fail2ban uses iptables to ban?
| joana035 wrote:
| Fail2ban uses whatever one wants to do whatever one need
| based on logs pattern matching :-)
| Jnr wrote:
| This sounds like a complex port knocking setup with bigger
| overhead. If you want better security and UDP is not a problem,
| consider use of Wireguard VPN. It is passive and silent, random
| attackers won't even know it is there.
| cperciva wrote:
| I still think that running ssh over spiped is more secure. Every
| byte being passed to sshd is cryptographically verified (and
| encrypted).
| thatha7777 wrote:
| This is very similar to port knocking, although more complicated.
| Although a proper JWT token is "more secure" than a sequence of
| 0-65535 integers, I contend that having complicated and/or
| unvetted logic as your first line of defense is more problematic
| than secure.
| kbenson wrote:
| On the positive side, it's probably pretty easy to encrypt the
| RPC traffic. On the negative side, you've traded the fairly
| well known risk of running SSH with some extra set of software
| (whoops: s/hardware/software/) with less well known vetting,
| and it's own possible security problems. This is basically
| relying on security through obscurity because of that.
|
| SSH is almost definitely handled by your OS package management,
| so any problems with SSH will be patched immediately as found,
| and automatically distributed to you if you opt in to that (or
| you will be notified there are updated to apply, and you can do
| so). If this system is not packaged and distributed by your OS,
| you're going to have to keep up to date on any security
| problems with it yourself. It may have less permissions by
| default because it's running through a container, but that
| doesn't always help (kernel exploits are a thing, a lot of
| successful hacks are first getting the ability to run
| unprivileged arbitrary code, then using a kernel exploit to
| gain privileged access.
|
| There's something to be said for well engineered, well vetted
| simplicity. This is an interesting solution, but I would much
| rather just lock down the set of IPs it allows connections from
| to SSH to a very small subset of known good ones, including the
| IP address of a $5 DigitalOcean VM that I keep off but can turn
| on and connect to and use to jump to that box if at a location
| I can't connect from directly (and you can set a timed
| firewalld rules to allow access from your current IP if it's
| not permanent and shut of the VM again). Of use the free tier
| of AWS if you can get it to have a specific IP for free as well
| (I don't recall what you can get in the free tier other than
| the smallest VM).
|
| Edit: Oh, and disable password auth. That's SSH hardening
| action #1, I kind of just assume everyone does that (because
| they should, if they can).
| jrochkind1 wrote:
| Yeah, I'd want to see a mention of port-knocking in the README,
| indicating that the author was familiar with this technique,
| and the known critiques of it, and why they think this method
| (or port-knocking in general?) withstands it. Cause it's
| definitely in the vicinity.
|
| With no mention, it makes me wonder if the author is familiar
| with the history of this approach, which is not a good sign.
|
| And yeah, I'm skeptical. Is this new, uncommon/unvetted, and
| somewhat complex "access" app just another place for potential
| vulnerabilities that give someone the keys to the kingdom?
| Bnshsysjab wrote:
| Technically they could be equally as secure if the port
| knocking converts knocked ports into a signature based
| authentication mechanism ;)
|
| But at that point, you've potentially introduced exploitable
| bugs in the network stack.
___________________________________________________________________
(page generated 2020-09-03 23:00 UTC)