Tailscale SSH: Investigation and remediation update

(tailscale.com)

103 points | by figmert 1 hour ago

11 comments

  • minimaltom 10 minutes ago
    Former tailscalar here and now someone managing tailscale at an org.

    Having done my own review of the vulns and the fixes as they are landing in github, its pretty clear to me that this only affects situations where you use ACLs to permit ssh into a node (via tailscale ssh, not via opensshd), but dont give a path to sudoers/root.

    If a user had no SSH access and/or connectivity to a vulnerable node, none of these vulns give them access.

    However, if you have a vulnerable tailscaled and you give ssh access as some non-root account/privs, then yes, these vulns are relevant as theres now known privilege escalations that give access beyond some specific user account.

    Not trying to play down the vuln (its important), but its not a free-for-all RCE that some ppl are acting like it is.

  • hollow-moe 6 minutes ago
    Recently NetBird also had a huge privesc through their SSH component too... I'm using both at $DAYJOB and my personal infra, damn you guys ought to be more careful with such a sensible and important component...
  • rietta 11 minutes ago
    I am fairly glad I have stuck with stock OpenSSH that ships with Linux. This incident proves I am glad with that choice.

    Over the decades, I have maintained remote access through port forwarding on my router - at one point in 2008 I put SSH on port 80 so I could access it from the restricted wifi network at the hospital my grandmother was at. I have also used openvpn. I have also used cloudflare tunnels. But in the end, I always connect to hardened openssh with key only authentication.

  • wrxd 1 hour ago
    I like Tailascale as a product but I decided to move my homelab to plain WireGuard.

    I wasn't using any of the additional features, I can get away with exposing a port on my router and I am changing devices rarely.

    News like this make me happy I removed a lot of complexity and attack surface

    • femiagbabiaka 48 minutes ago
      > I can get away with exposing a port on my router

      using wireguard is awesome, but I wouldn't say deploying and maintaining it lacks complexity exactly. and you didn't get rid of attack surface, you just made it your responsibility ;)

      • ectospheno 29 minutes ago
        Always is your responsibility even with tailscale. Now it is his responsibility without tailscale.
      • swingandamiss 42 minutes ago
        With AI, the complexity doesn't really matter anymore. Codex or Claude can easily configure complex site-to-site setups and peer configurations on mobile devices. I did this by hand before and it was hard and difficult to get right. Now I just tell AI and it does so perfectly, checks and verifies things. AI made life so easy.
        • drum55 34 minutes ago
          No amount of claude can handle the key thing that Tailscale fixes about wireguard, all of the connection to connection logic is handled be it direct, via all sorts of weird hole punching or via relays. Wireguard itself is horrible to do this with, half the time you're just looking at `wg status` wondering why the packet counter isn't incrementing today. It can never achieve the magic that is a direct connection between plane wifi and a host that's behind NAT on the ground.
        • arcanemachiner 37 minutes ago
          This is true, but I still wouldn't discount the possibility of mistakes. It's good to know how your home network works. Even Opus 5.5 can make some pretty dumb mistakes, particularly if it doesn't have the full picture.
          • swingandamiss 26 minutes ago
            Humans also make mistakes. At this point, I trust AI more than humans when it comes to casually configuring VPNs. I've read many who exposed things by mistake on reddit. AI has nearly solved this, probably is a solved issue at this point.
        • femiagbabiaka 37 minutes ago
          I would not personally expose my devices over a VPN that I self-hosted and set up purely through AI unless I felt like I understood how it worked completely. But that's probably just personal paranoia.
          • swingandamiss 24 minutes ago
            Just go read reddit threads on people who do this by hand following half baked guides. There's a huge population doing this, now everyone is switching to AI. It's a big win and safer. Not everyone is a trained network engineer, but now they are with AI
      • LoganDark 38 minutes ago
        Attack surface almost entirely goes away if you just require a good port knock first! I do that with my protection SSH servers and it works just fine. Also prevents anyone from knowing there even is an SSH server there. Or any server if it doesn't respond to pings either, and only uses reverse tunnels to accept public connections
        • neilalexander 16 minutes ago
          Wireguard doesn’t respond to malformed packets or handshakes from unknown devices by design.
        • mbreese 34 minutes ago
          Is there a way to automate port knocks in your .ssh/config? I think that’s the main reason why I haven’t added it to my home server.
          • LoganDark 19 minutes ago
            Technically you could use a ProxyCommand that does the knock and then runs nc to your target.
        • lakeviewterrace 33 minutes ago
          Port knocking is idiotic and amounts to Biff Tannen's "only I know how to start this car".
          • Gigachad 19 minutes ago
            Dunno, in combination with having an up to date system following best practices, port knocking is probably going to save you when bots start scanning the internet with the next SSH 0day. Any amount of friction will cause the bots to move to the next IP address.
            • icedchai 14 minutes ago
              Just moving SSH off a default port gets you 98% there.
              • LoganDark 11 minutes ago
                An attacker only has to get lucky once, so while I do put SSH on a non-default port too, it's not enough to avoid attacks on its own.
            • LoganDark 13 minutes ago
              It can also save you if, theoretically, there's ever a public exploit going around that can achieve RCE on nearly all versions of SSH, and some moron decides that nearly a day of embargo on the patch is more important. And said moron seemingly wasn't even trying to make sure Linux distros had security updates ready either, they were just sitting on it while the exploit was at large. This is apparently a risk now.
            • esseph 16 minutes ago
              LLMs port knock for redteam scans. Have for a long time.
              • LoganDark 2 minutes ago
                My knock for SSH was 20 ports within 2 seconds.
    • fooqux 1 hour ago
      From the sounds of what info they felt was safe to provide, this wouldn't have impacted you anyway.
      • giwook 49 minutes ago
        Don’t think that refutes their point.
    • F_r_k 24 minutes ago
      BTW, netbird is a cool alternative also
  • slowin 1 hour ago
    I self-host a tailnet with Headscale and could not be happier: https://headscale.net/stable/
  • 6thbit 1 hour ago

      > Most of the Tailscale SSH issues we've identified concern privilege escalation. In some circumstances, someone already authorized to connect over Tailscale SSH may be able to perform actions beyond the permissions of their configured local user, including potentially gaining elevated privileges.
    
    
    "Most"? "In some circumstances"? Makes me think that they could also be investigating a full on unauthorized connection issue, or am I reading too much into it?
    • remus 1 hour ago
      I think they're just writing cautiously because they're midway through the investigation and don't want to give false confidence. Doesn't seem worth reading too much into it until they provide a more thorough write up.
    • awildfivreld 1 hour ago
      The underlying writeup can be found at https://tailworm.com/
      • drum55 41 minutes ago
        10,000 words of claudish, it's not worth reading.
    • pizzafeelsright 1 hour ago
      Obviously tailscale has some great devs, security is complex.

      There is some weaselly language.

      In some circumstances default config? with password? this is a condition leading to the next

      someone already authorized to connect over Tailscale SSH they have access, but it may be limited or restricted, although scope could be beyond the intended

      may be able to perform actions beyond the permissions If there is a full moon and your brother in law's second cousin is wearing a hat, he may be able to use your upstair's thermostat provided you have a second story.

      • traceroute66 15 minutes ago
        > Obviously tailscale has some great devs, security is complex.

        Quick reminder that in recent months Tailscale has had issues with age-old classic vulnerabilities such as input validation.

        The problem with Tailscale is twofold.

        First their attack surface is too big. They continuously shoehorn new features into the product.

        Second they don't appear to do regular deep-dive code audits and instead they appear to rely mostly on the goodwill of third-parties to poke at select areas of their code as and when those third-parties feel like it. Hence simple vulnerabilities such as input validation keep on popping up.

        Fancy SOC2 certificates are not the same thing. That's just dogfood for the corporate auditors.

        At $work we are busy working on removing Tailscale. The bells and whistles of Tailscale sure are attractive candy, but we simply can't overlook the security risk of what has become a bloated product.

      • fooqux 27 minutes ago
        Given what just happened with PHP a few days so, it's not surprising to me that they're holding their cards close to their chest. People will already be doing their damnedest to reverse engineer the exploit with this meager info.
    • devmor 1 hour ago
      A security researcher colleague of mine was/is exploring an unauthenticated avenue along this line. I don’t have any update, she told me she didn’t feel comfortable exposing information to anyone other than Tailscale staff until she’s confident in what she found or didn’t.
      • raggi 1 minute ago
        we'd love to hear about it at security@tailscale.com, we're happy to work from & with and credit early ideas as well.
  • aborsy 1 hour ago
    There are too many features, and lines of code (probably more than OpenVPN). I'm pretty sure there are all kinds of zero days, with this attack surface and feature set.

    I use it extensively, and I would rather pay them a fee if they could review thr code.

    Another possibility is as lock down mode, disabling fancy and risky features.

    • drdexebtjl 16 minutes ago
      This is also how I feel about Tailscale and its popular alternatives. They do too much.

      Does anyone know of a Tailscale alternative that sticks to the basics?

      Something that’s just WireGuard but without the manual key exchange, and with Tailscale-like NAT hole-punching that falls back to using an oblivious relay?

  • MisterTea 40 minutes ago
    Not a tailscale user so I am a bit lost here: where does SSH fit into a Wireguard based VPN service?
  • Gigachad 14 minutes ago
    Everyone is twitchy over security right now, but at first glance this seems not that bad. If the attacker has to be manually added to your tailnet that makes this not very exploitable. You should be more worried about all the iot slop on your local network.

    Good idea to set up automatic updates for the tailscale client though.

  • wslh 1 hour ago
    With the success of Tailscale I would like to be sure that their repositories and scenarios are continuously reviewed. I understand that there is always a next hop in cybersecurity though.
  • _def 1 hour ago
    I think it's so invasive that they ship this per default. But thats the business model i guess.
    • seemaze 1 hour ago
      I've always had to manually enable this feature. I've never experienced it on by default across multiple platforms and deployments.
      • RobMurray 0 minutes ago
        I've never used it once. don't even see the point. normal ssh works fine over Tailscale.

        What is the advantage? Why would I want to swap the well tested auth in ssh for auth managed by tailscale?

    • femiagbabiaka 1 hour ago
      Do they ship this by default? I thought it had to be enabled: https://tailscale.com/docs/features/tailscale-ssh#configure-...
    • forbiddenlake 1 hour ago
      How do you mean? You have to run a command to enable tailscale ssh, it isn't on by default.
    • LeoPanthera 1 hour ago
      Tailscale SSH is not enabled by default.
      • zer0zzz 31 minutes ago
        It’s also not even available on some platforms afaik