Great! This feature made a lot of sense, and it took a long time.
It’s like falling back to hub and spoke, except that the traffic is end to end encrypted, and the middle node is used only when direct connection is not possible, and for some clients. It’s also similar to running your own derp server (which works also in TCP), but without the hassle of doing so, and perhaps without having to open ports to the internet (needed in derp) so long as the relay is reachable by peers.
The derp servers have low throughput. Another option could be a pay-as-you-go derp service.
They might also be on their way to remove the need for reverse proxies, with the recent announcement on Tailscale services.
BTW, why could it be paid for more than two relays? You are using just your own devices and bandwidth :)
It actually lower the bandwidth bill for Tailscale by reducing the usage of their own relays. Ideally, by default the software will find whatever nodes could help with direct connection. It’s just routing within your own network.
> It’s also similar to running your own derp server (which works also in TCP), but without the hassle of doing so, and perhaps without having to open ports to the internet (needed in derp) so long as the relay is reachable by peers.
I think most folks will need to open a port to the internet, because otherwise you wouldn't need the tailscale to begin with. e.g. connecting your cloud network to your on premise network etc.
Of course exceptions apply, like both clients can reach the peer relay, but not each other directly.
I could open a port to the internet, but it would be Tailscale’s responsibility to secure the software that listens to the port (subject to an up-to-date software, that is my responsibility).
It’s not a standard Wireguard port. With Wireguard included in Linux, I would not be worried.
This was better solved by tinc about 20 years ago. All tinc nodes can work as relays (but you can disallow that if you want), it does not rely on a centralized server, and works fine without access to the internet. It is a true mesh. The world would be better served by porting tinc to wireguard and some memory safe language instead of reimplementing parts of its functionality from scratch.
Heh. I have a 30-nodes Tinc network over the internet but some hosts are behind a NAT. It keeps randomly losing routes between these nodes. It even has the infuriating behavior that often it loses the route a few seconds after I successfully established a SSH connection.
Also, traffic seems to be decrypted and re-encrypted by relaying nodes. For end-to-end encryption, you need "ExperimentalProtocol = yes" added by Tinc 1.1, which was never formally released.
I'd like to rewrite something like it in a language I'm familiar with (perhaps based on cjdns' protocol which is better documented than Tinc's) but it's not easy.
Wireguard can't punch through NATs or firewalls without third party software like Tailscale. Also I'm pretty sure each peer to peer connection needs to be individually set up in a config file ahead of time
Nebula[0] addresses this and is IMO an improvement over WireGuard. Came out of Slack originally, and it supports peer discovery, NAT hole punching, and some other cool features. Also still uses the Noise Protocol.
In practice, the extra networking features + better first class peer config management baked in is very nice (Nebula’s “lighthouses” are configured with a tool similar to DSNet for Wireguard[1])
I wonder if the next step could be to have all tailscale clients automatically able to accept forwarding requests between any two machines within the tailnet, so that the mesh seamlessly auto-routes around any breaks within the mesh?
Yeah, such extension is natural and it comes up frequently in the context of overlay networks. The defining question here is if you are OK with relaying other people's CP traffic.
One thing I didn’t understand: it uses an UDP port of my choice. What IP is it using? Everything via the tailnet or do I need to open this port to the internet?
If only available via Tailscale/tailnet - how is connectivity better since if two devices can connect to each other via Tailscale we are already on the direct connection route instead of a relay / derp connection?!
> It allows customers to make just one firewall exception for connections only coming from their tailnet.
You'll need to open a single UDP port on your firewall, so it's your public facing IP address. You don't need an entire VM somewhere, just a single port.
Regarding the speed question. You'd use the derp when it's not possible to make a peer to peer connection, which limits your speed to derp server's speed and load. Which the peer relay, you can practically use the entire bandwidth you have between your devices.
So instead of whitelisting all ports from IP range 100.64.0.0/10 I would just whitelist e.g. UDP port 12345 coming from IP range 100.64.0.0/10 to my public IP? Or just open up UDP 12345 completely?
What's the use case for this? It seems to be for situations where you might have a SaaS product, but there is some data required from a customer system. You'd expose the customer data using this relay and integrate into the SaaS. Is that the gist of it? Integration would still likely involve you giving the customer some software to expose a limited API and handle auth, logging, etc.
They are an alternative to the tailscale operated DERP servers, which are cloud relays.
Even with the much touted NAT punching capabilities of tailscale, there are numerous instances where tailscale cannot establish a true p2p connection. The last fallback is the quite slow DERP relay and from experience it gets used very often.
If you have a peer in your tailscale network that has a good connection and that maybe you can even expose to the internet with a port forward on your router, you now have this relay setting that you can enable to avoid using the congested/shared DERP servers. So there is not really a new use-case for this. It's the same, just faster.
The explanation that I think wasn't entirely clear in the post is how it actually works/why that's better.
From what I can tell, the situation is this:
1. You have a host behind NAT
2. That NAT will not allow you to open ports via e.g. uPnP (because it's a corporate firewall or something, for example) so other tailscale nodes cannot connect to it
3. You have another host which has the same configuration, so neither host can open ports for the other to connect in
The solution is to run a peer relay, which seems to be another (or an existing) tailscale node which both of these hosts can connect to via UDP, so in this circumstance it could be a third node you're already running or a new one you configure on a separate network.
When the two NAT'ed hosts can't connect to each other, they can both opt to connect instead to this peer node allowing them to communicate with each other via the peer node.
Previously this was done via Tailscale's hosted DERP nodes; these nodes would facilitate tailscale nodes to find each other but could also proxy traffic in this hard-NAT circumstance. Now you can use your own node to do so, which means you can position it somewhere that is more efficient for these two nodes to connect to and where you have control over the network, the bandwidth, the traffic, etc.
Tailscale is a few things. It might be fair to say that it is mostly a software platform with a web frontend that allows orgs (and individual users alike) to easily create secure VPNs, so their various systems can have a secure, unfiltered virtual private network on which to communicate with eachother even if they're individually scattered across the four corners of the Internet.
The usual (traditional) way to do VPN stuff is/was hub-and-spoke: Each system connected to a central hub, and through that hub each system had access to the other systems.
But the way that Tailscale operates is different than that: Ideally, each connected system forms a direct UDP/IP connection with every other system on the VPN. There is no hub. In this way: If node A has data to send to node F, then it can send it directly there without traversing through a central hub.
And that's pretty cool -- this peer-to-peer arrangement is gloriously efficient compared to hub-and-spoke. (It's efficient enough that a person can get quite a lot done with Tailscale for free, with no payment expected ever.)
But we don't live in an ideal world. We instead often live in a world of NAT and firewalls -- sometimes even implemented by the ISPs themselves -- that can make it impossible for two nodes to directly send UDP packets to eachother. This results in unreachable nodes, which is not useful.
So Tailscale's workaround to that Internet problem is to provide Designated Encrypted Relays for Packets (DERP). DERP usually works, and end-to-end encryption is maintained.
DERP is also not at all new. It brings back some aspects of hub-and-spoke, but only for nodes that can't communicate directly; DERP behaves in a way akin to a hub, to help these crippled nodes by relaying traffic between them and the rest of the VPN's nodes.
But DERP is a Tailscale-hosted operation. And it can be pretty slow for some applications. And there was no way, previously, for an individual user to improve the performance of DERP: It simply was whatever it was -- with a collection of DERP servers chewing through bandwidth to provide connectivity for a world of badly-connected VPN nodes.
But today's announcement brings forth Tailscale Peer Relay.
> What's the use case for this?
The primary use case for this is simple: It is an alternative to DERP. A user can now provide their own relay service for their network's badly-connected peers to use. So now, rather than being limited to whatever bandwidth DERP has available, relaying can offer as much bandwidth as a user can afford to pay for and host themselves.
And if a user plans it right, then they can put their Peer Relay somewhere on the network where it can help minimize inter-node latency compared to DERP.
(It's not for everyone. Tailscale isn't for everyone, either -- not everyone needs a VPN at all. I'd never expect a random public customer to use it knowingly and directly.)
Hard to parse the networking jargon, but does this enable offline connections?
If I have two devices on my local LAN (both connected to a Tailnet) and my home internet goes, currently the devices disconnect from each other. I have been looking for a way to prevent that, so that the all devices connected to the same WiFi network on a tailnet can find each other even if the internet connection to the wider world is broken.
> my home internet goes, currently the devices disconnect from each other.
You might be able to solve this by hosting your own control plane such as Headscale. Instead of having Tailscale manage/coordinate all the nodes on tailnet.
That'd be easy if only wireguard supported multiple endpoints per peer, then one could add the ULA address of the local devices in addition to the external one.
How does this interact with machines that are shared from one Tailnet to another? Is there specific syntax to grant the appropriate permission to a user or device that accesses the destination via sharing?
The docs also say:
> As a rule of thumb, the src devices in the grant policy should typically be devices in a stable physical location behind a strict NAT or firewall that prevents direct connections. This typically includes devices in corporate networks or cloud environments. It usually does not include mobile devices or laptops that frequently change locations and network conditions.
Is there some reason that one should not set up a peer relay to enable a laptop to access a machine that is behind a NAT? (Tailscale regularly fails to establish direct connectivity from a laptop behind a NAT to a machine that's behind a different NAT, at least in my experience.)
One limitation of custom DERP is that across tailnets, they don't share the same DERP maps and don't have access to each others' DERPs.
With Tailscale Peer Relays, the available relay bindings can be seen by the devices on either side of a connection; as such it should work out of the box with a sharing relationship between tailnets.
In your example the src would be the "machine that is behind a NAT". That's the one the peer relay enable access to. And then all your other devices (that laptop) can reach it through the peer relay.
I was also a bit confused on the meaning of src/dst in the grants. The naming didn't match my thinking.
My reading of the docs suggests that the relays and both peers must be in the same tailnet. Additionally, both peers must have the correct ACLs set up to access each other
How to do site-to-site traffic over Tailscale / WG encryption? From preliminary testing, it seems have difficulty to saturate a 10Gbps connection while plain HTTP (nginx) traffic does that fine. Of course it should vary from CPU to CPU, but any tips how to improve that? Ideally I would love to go over with encrypted traffic, although everything is public, just one less thing need to be careful (in case future need to transport some non-public data over).
> All customers can use two peer relays, for free, forever. As your needs scale, so will the number of available peer relays. To add even more peer relays to your tailnet, come have a chat with us.
I have to pay to be able to donate my own infra to make tailscale's service better?
I kinda doubt we'll end up charging for it (as it costs us ~nothing except support costs, which are real), but it's easier to make it free later when it's GA rather than rug pull on people and start charging for it in the future if we start it out free+unlimited.
While I don't think that's accurate, I, too, am surprised that this feature isn't¹ completely free. After all, it will make it easier (or possible at all) for many people to use Tailscale.
Great! This feature made a lot of sense, and it took a long time.
It’s like falling back to hub and spoke, except that the traffic is end to end encrypted, and the middle node is used only when direct connection is not possible, and for some clients. It’s also similar to running your own derp server (which works also in TCP), but without the hassle of doing so, and perhaps without having to open ports to the internet (needed in derp) so long as the relay is reachable by peers.
The derp servers have low throughput. Another option could be a pay-as-you-go derp service.
They might also be on their way to remove the need for reverse proxies, with the recent announcement on Tailscale services.
BTW, why could it be paid for more than two relays? You are using just your own devices and bandwidth :)
It actually lower the bandwidth bill for Tailscale by reducing the usage of their own relays. Ideally, by default the software will find whatever nodes could help with direct connection. It’s just routing within your own network.
> It’s also similar to running your own derp server (which works also in TCP), but without the hassle of doing so, and perhaps without having to open ports to the internet (needed in derp) so long as the relay is reachable by peers.
I think most folks will need to open a port to the internet, because otherwise you wouldn't need the tailscale to begin with. e.g. connecting your cloud network to your on premise network etc.
Of course exceptions apply, like both clients can reach the peer relay, but not each other directly.
I could open a port to the internet, but it would be Tailscale’s responsibility to secure the software that listens to the port (subject to an up-to-date software, that is my responsibility).
It’s not a standard Wireguard port. With Wireguard included in Linux, I would not be worried.
This was better solved by tinc about 20 years ago. All tinc nodes can work as relays (but you can disallow that if you want), it does not rely on a centralized server, and works fine without access to the internet. It is a true mesh. The world would be better served by porting tinc to wireguard and some memory safe language instead of reimplementing parts of its functionality from scratch.
Heh. I have a 30-nodes Tinc network over the internet but some hosts are behind a NAT. It keeps randomly losing routes between these nodes. It even has the infuriating behavior that often it loses the route a few seconds after I successfully established a SSH connection.
Also, traffic seems to be decrypted and re-encrypted by relaying nodes. For end-to-end encryption, you need "ExperimentalProtocol = yes" added by Tinc 1.1, which was never formally released.
I'd like to rewrite something like it in a language I'm familiar with (perhaps based on cjdns' protocol which is better documented than Tinc's) but it's not easy.
Wireguard can do what you're describing.
Wireguard can't punch through NATs or firewalls without third party software like Tailscale. Also I'm pretty sure each peer to peer connection needs to be individually set up in a config file ahead of time
Nebula[0] addresses this and is IMO an improvement over WireGuard. Came out of Slack originally, and it supports peer discovery, NAT hole punching, and some other cool features. Also still uses the Noise Protocol.
In practice, the extra networking features + better first class peer config management baked in is very nice (Nebula’s “lighthouses” are configured with a tool similar to DSNet for Wireguard[1])
[0] https://github.com/slackhq/nebula [1] https://github.com/naggie/dsnet
So now we're back "tailscale but with different steps"
Folks have solved these problems and open sourced solutions: https://github.com/tonarino/innernet
But all that I've seen are still centralized/federated
Neither can tinc.
... because its an problem with NAT and not with the protocol.
A $2.5/month vps solves this issue.
I wonder if the next step could be to have all tailscale clients automatically able to accept forwarding requests between any two machines within the tailnet, so that the mesh seamlessly auto-routes around any breaks within the mesh?
Yeah, such extension is natural and it comes up frequently in the context of overlay networks. The defining question here is if you are OK with relaying other people's CP traffic.
"within the tailnet." So, the "other people" are people you've authorized to be on your tailnet, at least in this conversation.
Any reason this is not supported on iOS/tvOS devices? Would love to make it work on my Apple TV!
I'm paraphrasing what one of the engineers said when I asked this question to the team (I'm a TS insider):
There's an application size limit associated with NetworkExtension.
https://developer.apple.com/documentation/networkextension
So they would need to get the binary size smaller in order to implement this on iOS or Apple TV.
One thing I didn’t understand: it uses an UDP port of my choice. What IP is it using? Everything via the tailnet or do I need to open this port to the internet?
If only available via Tailscale/tailnet - how is connectivity better since if two devices can connect to each other via Tailscale we are already on the direct connection route instead of a relay / derp connection?!
> It allows customers to make just one firewall exception for connections only coming from their tailnet.
You'll need to open a single UDP port on your firewall, so it's your public facing IP address. You don't need an entire VM somewhere, just a single port.
Regarding the speed question. You'd use the derp when it's not possible to make a peer to peer connection, which limits your speed to derp server's speed and load. Which the peer relay, you can practically use the entire bandwidth you have between your devices.
> for connections coming from their tailnet
So instead of whitelisting all ports from IP range 100.64.0.0/10 I would just whitelist e.g. UDP port 12345 coming from IP range 100.64.0.0/10 to my public IP? Or just open up UDP 12345 completely?
What's the use case for this? It seems to be for situations where you might have a SaaS product, but there is some data required from a customer system. You'd expose the customer data using this relay and integrate into the SaaS. Is that the gist of it? Integration would still likely involve you giving the customer some software to expose a limited API and handle auth, logging, etc.
They are an alternative to the tailscale operated DERP servers, which are cloud relays.
Even with the much touted NAT punching capabilities of tailscale, there are numerous instances where tailscale cannot establish a true p2p connection. The last fallback is the quite slow DERP relay and from experience it gets used very often.
If you have a peer in your tailscale network that has a good connection and that maybe you can even expose to the internet with a port forward on your router, you now have this relay setting that you can enable to avoid using the congested/shared DERP servers. So there is not really a new use-case for this. It's the same, just faster.
The explanation that I think wasn't entirely clear in the post is how it actually works/why that's better.
From what I can tell, the situation is this:
1. You have a host behind NAT
2. That NAT will not allow you to open ports via e.g. uPnP (because it's a corporate firewall or something, for example) so other tailscale nodes cannot connect to it
3. You have another host which has the same configuration, so neither host can open ports for the other to connect in
The solution is to run a peer relay, which seems to be another (or an existing) tailscale node which both of these hosts can connect to via UDP, so in this circumstance it could be a third node you're already running or a new one you configure on a separate network.
When the two NAT'ed hosts can't connect to each other, they can both opt to connect instead to this peer node allowing them to communicate with each other via the peer node.
Previously this was done via Tailscale's hosted DERP nodes; these nodes would facilitate tailscale nodes to find each other but could also proxy traffic in this hard-NAT circumstance. Now you can use your own node to do so, which means you can position it somewhere that is more efficient for these two nodes to connect to and where you have control over the network, the bandwidth, the traffic, etc.
Tailscale is a few things. It might be fair to say that it is mostly a software platform with a web frontend that allows orgs (and individual users alike) to easily create secure VPNs, so their various systems can have a secure, unfiltered virtual private network on which to communicate with eachother even if they're individually scattered across the four corners of the Internet.
The usual (traditional) way to do VPN stuff is/was hub-and-spoke: Each system connected to a central hub, and through that hub each system had access to the other systems.
But the way that Tailscale operates is different than that: Ideally, each connected system forms a direct UDP/IP connection with every other system on the VPN. There is no hub. In this way: If node A has data to send to node F, then it can send it directly there without traversing through a central hub.
And that's pretty cool -- this peer-to-peer arrangement is gloriously efficient compared to hub-and-spoke. (It's efficient enough that a person can get quite a lot done with Tailscale for free, with no payment expected ever.)
But we don't live in an ideal world. We instead often live in a world of NAT and firewalls -- sometimes even implemented by the ISPs themselves -- that can make it impossible for two nodes to directly send UDP packets to eachother. This results in unreachable nodes, which is not useful.
So Tailscale's workaround to that Internet problem is to provide Designated Encrypted Relays for Packets (DERP). DERP usually works, and end-to-end encryption is maintained.
DERP is also not at all new. It brings back some aspects of hub-and-spoke, but only for nodes that can't communicate directly; DERP behaves in a way akin to a hub, to help these crippled nodes by relaying traffic between them and the rest of the VPN's nodes.
But DERP is a Tailscale-hosted operation. And it can be pretty slow for some applications. And there was no way, previously, for an individual user to improve the performance of DERP: It simply was whatever it was -- with a collection of DERP servers chewing through bandwidth to provide connectivity for a world of badly-connected VPN nodes.
But today's announcement brings forth Tailscale Peer Relay.
> What's the use case for this?
The primary use case for this is simple: It is an alternative to DERP. A user can now provide their own relay service for their network's badly-connected peers to use. So now, rather than being limited to whatever bandwidth DERP has available, relaying can offer as much bandwidth as a user can afford to pay for and host themselves.
And if a user plans it right, then they can put their Peer Relay somewhere on the network where it can help minimize inter-node latency compared to DERP.
(It's not for everyone. Tailscale isn't for everyone, either -- not everyone needs a VPN at all. I'd never expect a random public customer to use it knowingly and directly.)
I was literally looking for a solution for this over the weekend and ended up with a very quirky setup for my Kubernetes Operator.
Now I can rip all that out and use this! Bravo!
Man K8s is my nightmare haha. 100% agree
Hard to parse the networking jargon, but does this enable offline connections?
If I have two devices on my local LAN (both connected to a Tailnet) and my home internet goes, currently the devices disconnect from each other. I have been looking for a way to prevent that, so that the all devices connected to the same WiFi network on a tailnet can find each other even if the internet connection to the wider world is broken.
> my home internet goes, currently the devices disconnect from each other.
You might be able to solve this by hosting your own control plane such as Headscale. Instead of having Tailscale manage/coordinate all the nodes on tailnet.
https://github.com/juanfont/headscale
I ran headscale for a while and can confirm local devices could still chat when the internet went down.
That'd be easy if only wireguard supported multiple endpoints per peer, then one could add the ULA address of the local devices in addition to the external one.
that already works
Not for me. Maybe some misconfiguration on my part then.
run `tailscale status` and ensure that your local machines are connected to each other `direct` and not using a relay.
Yay! Excited to see them building this in public.
I recall that tailscale DERP servers were always slow and made things feel delayed when they had to be used as a relay.
Why go through the effort of reimplementing all this instead of using libp2p?
You mean the library that almost got me banned from my VPS provider because it dialed about half the RFC1918 space?
I can finally tear down my custom DERP server that I was using to get higher throughput between two NAT'd clients.
How does this interact with machines that are shared from one Tailnet to another? Is there specific syntax to grant the appropriate permission to a user or device that accesses the destination via sharing?
The docs also say:
> As a rule of thumb, the src devices in the grant policy should typically be devices in a stable physical location behind a strict NAT or firewall that prevents direct connections. This typically includes devices in corporate networks or cloud environments. It usually does not include mobile devices or laptops that frequently change locations and network conditions.
Is there some reason that one should not set up a peer relay to enable a laptop to access a machine that is behind a NAT? (Tailscale regularly fails to establish direct connectivity from a laptop behind a NAT to a machine that's behind a different NAT, at least in my experience.)
One limitation of custom DERP is that across tailnets, they don't share the same DERP maps and don't have access to each others' DERPs.
With Tailscale Peer Relays, the available relay bindings can be seen by the devices on either side of a connection; as such it should work out of the box with a sharing relationship between tailnets.
In your example the src would be the "machine that is behind a NAT". That's the one the peer relay enable access to. And then all your other devices (that laptop) can reach it through the peer relay.
I was also a bit confused on the meaning of src/dst in the grants. The naming didn't match my thinking.
Hmm. It would be very nice if this worked when the laptop is on a different tailnet.
My reading of the docs suggests that the relays and both peers must be in the same tailnet. Additionally, both peers must have the correct ACLs set up to access each other
How to do site-to-site traffic over Tailscale / WG encryption? From preliminary testing, it seems have difficulty to saturate a 10Gbps connection while plain HTTP (nginx) traffic does that fine. Of course it should vary from CPU to CPU, but any tips how to improve that? Ideally I would love to go over with encrypted traffic, although everything is public, just one less thing need to be careful (in case future need to transport some non-public data over).
You'd need to analyze what's bottlenecking when you're doing that.
Even a cursory look at htop on both ends while you're trying to saturate that link would be informative.
OK, another use for port 1337. Anyone got a better, mnemonically memorable choice in the low range?
I'm using 3377 (D-E-R-P) since it's effectively replacing the DERP servers on the specified source devices.
> All customers can use two peer relays, for free, forever. As your needs scale, so will the number of available peer relays. To add even more peer relays to your tailnet, come have a chat with us.
I have to pay to be able to donate my own infra to make tailscale's service better?
Tailscale employee here.
I kinda doubt we'll end up charging for it (as it costs us ~nothing except support costs, which are real), but it's easier to make it free later when it's GA rather than rug pull on people and start charging for it in the future if we start it out free+unlimited.
> to make tailscale's server better?
While I don't think that's accurate, I, too, am surprised that this feature isn't¹ completely free. After all, it will make it easier (or possible at all) for many people to use Tailscale.
¹) s/isn't/might not end up being. See https://news.ycombinator.com/item?id=45751253
[flagged]
[flagged]