While most engineers use SSH exclusively for shell access, its true power lies in its ability to act as a secure, arbitrary TCP tunnel. By leveraging local, remote, and dynamic port forwarding, you can bypass the need for complex VPNs or risky public IP exposures for internal services like databases, monitoring dashboards, and staging environments.
Why This Matters
In production environments, the surface area for attacks is often expanded by “temporary” holes poked in firewalls for debugging. I’ve seen countless incidents where a developer opened a RDS instance to 0.0.0.0/0 just to run a manual migration, only to forget it for three months.
Traditional VPNs are often heavy, brittle, and require complex client-side configuration. SSH tunnels provide a lightweight, ephemeral alternative that uses the existing identity and access management (IAM) of your bastion hosts. It allows you to treat the network as hostile while maintaining the ability to reach private resources without exposing them to the public internet.
Architecture / System Design
SSH tunneling operates at the application layer but behaves like a transport-level proxy. It wraps TCP packets inside the encrypted SSH stream, effectively extending your local networking stack into the remote VPC.
There are three primary primitives you need to manage:
- Local Forwarding (-L): Pulls a remote service to your local machine. Think: accessing a private Postgres instance on localhost:5432.
- Remote Forwarding (-R): Pushes a local service to a remote host. Think: exposing a local webhooks listener to a staging server.
- Dynamic Forwarding (-D): Turns the SSH connection into a SOCKS5 proxy. This is the most powerful for high-cardinality environments where you need to reach dozens of internal microservices without mapping individual ports.
When we scale this, we move away from raw flags and into ProxyJump architectures. Instead of a single hop, the traffic traverses multiple security zones—from your laptop, through a hardened bastion, and finally to the target service—all over a single, encrypted pipe.
Implementation
For a standard production debugging scenario where you need to reach a private database, the direct approach is often the most reliable.
# Mapping a remote RDS instance to local port 5432 via a bastion host
ssh -L 5432:db-internal.cluster.us-east-1.rds.amazonaws.com:5432 user@bastion-host.example.com
However, for engineering teams, managing these manually is a recipe for configuration drift. We standardize this via the ~/.ssh/config to abstract the infrastructure complexity.
Host production-db
HostName bastion.prod.example.com
User sahil
LocalForward 5432 internal-db-host:5432
ExitOnForwardFailure yes
ServerAliveInterval 60
ProxyJump jump-host.prod.example.com
If you are debugging a complex web of internal services (Prometheus, Grafana, HashiCorp Vault), dynamic forwarding is cleaner. It prevents port collisions on your local machine.
# Create a SOCKS5 proxy on port 1080
ssh -D 1080 -N -f user@bastion-host.example.com
Operational Realities
Operating SSH tunnels at scale introduces specific performance constraints. The most common is the “TCP-over-TCP” problem. Because SSH is a TCP-based protocol, wrapping another TCP stream inside it means you have two independent congestion control algorithms fighting each other.
On high-latency links or lossy connections, this results in a “TCP meltdown.” If a packet is lost in the outer SSH tunnel, it triggers a retransmission. Simultaneously, the inner application (the DB client) thinks the packet is lost and triggers its own retransmission. This can lead to exponential backoff and total throughput collapse.
From an observability perspective, SSH tunnels are often “dark traffic.” Standard VPC flow logs will only show traffic on port 22. To understand what’s actually happening, you need to monitor SSH session metadata or use an eBPF-based agent to inspect the underlying socket activity on the bastion host.
Failure Modes / Trade-offs
One major risk is the “Eternal Tunnel.” Engineers often script these tunnels to start on boot or via systemd. If the bastion host is compromised, these persistent tunnels provide an always-on bridge directly into your private subnets.
Another failure mode is resource exhaustion on the bastion. Each tunnel consumes a file descriptor. If you have a large team all tunneling their local environments through a single small t3.micro instance, you will hit MaxSessions or MaxStartups limits, causing new connections to drop silently.
There is also the “Leaky Proxy” risk with dynamic forwarding. If your browser or tool isn’t configured to resolve DNS through the SOCKS proxy, you may leak internal hostnames via your local DNS provider, even if the actual traffic is encrypted.
Lessons Learned / Best Practices
- Always use ExitOnForwardFailure yes in your configs to prevent silent failures where the SSH session stays open but the tunnel is dead.
- Use ServerAliveInterval to keep the connection through stateful firewalls/NATs that would otherwise drop idle connections.
- Limit bastion access using Match blocks in /etc/ssh/sshd_config to restrict which users are allowed to perform port forwarding.
- Implement ProxyJump instead of manually nesting SSH commands to reduce overhead and simplify the identity chain.
TL;DR
- SSH isn’t just for shells; it’s a production-grade TCP proxy for secure, ephemeral access.
- Use -L for specific services and -D (SOCKS5) for broad internal network access.
- Avoid “TCP Meltdown” by being mindful of latency when nesting TCP streams.
- Centralize tunnel configuration in ~/.ssh/config to reduce manual operational errors.
- Monitor bastion file descriptors and session limits to prevent scaling bottlenecks.
- Always set ServerAliveInterval to prevent firewall-induced connection drops.
Final Thoughts
Infrastructure is often built with the assumption that secure access requires heavy-handed networking changes. SSH tunnels prove that if you have a secure identity layer and a reachable port 22, you already have a sophisticated VPN at your fingertips. Use it to keep your private subnets private.