Traces of thoughts

View on GitHub

Troubleshooting: kubectl “No Route to Host” on macOS

Symptom

kubectl get nodes (and other kubectl commands) suddenly fail with:

Unable to connect to the server: dial tcp 10.0.0.226:6443: connect: no route to host

The cluster was previously accessible and no network or cluster configuration has changed.

Key Observations

Tool Result
ping <server-ip> Works
ssh user@<server-ip> Works
/usr/bin/curl -k https://<server-ip>:6443/api Works (returns 401 Unauthorized)
/usr/bin/nc -z <server-ip> 6443 Works
kubectl get nodes EHOSTUNREACH
helm list EHOSTUNREACH
Python socket.connect() EHOSTUNREACH
Compiled C connect() EHOSTUNREACH

The pattern: Apple-signed system binaries (/usr/bin/curl, /usr/bin/nc, /sbin/ping, /usr/bin/ssh) succeed, while all non-system binaries (Homebrew kubectl, Go programs, Python, compiled C) fail with errno 65 (EHOSTUNREACH).

Root Cause

macOS enforces Local Network privacy permissions on a per-application basis. When you run kubectl from a terminal emulator (iTerm2, Warp, VS Code, Ghostty, etc.), macOS applies the terminal app’s Local Network permission to all child processes.

Apple’s own system binaries in /usr/bin and /sbin have special entitlements (e.g., com.apple.security.network.client) that bypass this restriction. Homebrew-installed binaries and anything you compile yourself do not.

The permission can become stale or broken after:

When this happens, the Local Network toggle in System Settings may still show “on” but the actual permission is no longer effective. The terminal app’s sandbox profile (set at launch time) retains the broken state until the app is restarted with a fresh grant.

Diagnosis Steps

1. Confirm the cluster is reachable

ping -c 1 <server-ip>
curl -k https://<server-ip>:6443/api

If both succeed, the cluster and network are fine.

2. Check if the issue is binary-specific

# This will fail
kubectl get nodes

# This will succeed (Apple system binary)
/usr/bin/curl -k https://<server-ip>:6443/api

If curl works but kubectl doesn’t, the issue is macOS Local Network permissions.

3. Confirm it’s terminal-specific

Try running kubectl get nodes from Terminal.app (Apple’s built-in terminal). If it works there but not in your third-party terminal, you’ve confirmed the cause.

4. Rule out other causes

# Check no proxies are interfering
echo $HTTP_PROXY $HTTPS_PROXY $NO_PROXY

# Check macOS application firewall is not blocking
/usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate

# Check kubeconfig is correct
kubectl config view --minify

# Check client certificate validity
kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -dates

Fix

  1. Open System Settings > Privacy & Security > Local Network
  2. Find your terminal app (iTerm2, Warp, etc.)
  3. Toggle the permission OFF
  4. Toggle it back ON
  5. Quit the terminal app completely (Cmd+Q) and reopen it

The restart is required because the sandbox profile is applied at launch time – toggling the setting alone is not enough.

Why System Binaries Still Work

macOS treats binaries differently based on their code signature and entitlements:

This is why the same connect() syscall with identical parameters can return success for one binary and EHOSTUNREACH for another.

Affected Tools

Any non-system binary making local network TCP connections, including but not limited to:

Prevention