• OpenShift
  • ssh: unexpected packet in response to channel open

time="2025-04-16T17:46:45+09:00" level=debug msg="retry loop: attempt 9"
time="2025-04-16T17:46:45+09:00" level=debug msg="Running SSH command: <hidden>"
time="2025-04-16T17:46:45+09:00" level=debug msg="SSH command succeeded"
time="2025-04-16T17:46:45+09:00" level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
time="2025-04-16T17:46:47+09:00" level=debug msg="retry loop: attempt 10"
time="2025-04-16T17:46:47+09:00" level=debug msg="Running SSH command: <hidden>"
time="2025-04-16T17:46:47+09:00" level=debug msg="SSH command succeeded"
time="2025-04-16T17:46:47+09:00" level=debug msg="error: Temporary error: pull secret not updated to disk - sleeping 2s"
time="2025-04-16T17:46:49+09:00" level=debug msg="retry loop: attempt 11"
time="2025-04-16T17:46:49+09:00" level=debug msg="Running SSH command: <hidden>"
time="2025-04-16T17:47:15+09:00" level=debug msg="Failed to create new ssh session: ssh: unexpected packet in response to channel open: <nil>"
time="2025-04-16T17:47:15+09:00" level=debug msg="SSH command failed"
time="2025-04-16T17:47:15+09:00" level=debug msg="non-retriable error: failed to read /var/lib/kubelet/config.json file: ssh command error:\ncommand : sudo cat /var/lib/kubelet/config.json\nerr : ssh: unexpected packet in response to channel open: <nil>\n: "
time="2025-04-16T17:47:15+09:00" level=debug msg="Running 'sw_vers -productVersion'"

ps aux | grep vmnetd # Verified Docker network service
sudo launchctl bootout system /...vmnetd.plist # Disabled it for current session
sudo rm -f /Library/LaunchDaemons/...vmnetd.plist # Prevented it from auto-starting
crc start # โœ… CRC now boots and works properly!

ps aux | grep vmnetd # Verified Docker network service
sudo launchctl bootout system /...vmnetd.plist # Disabled it for current session
sudo rm -f /Library/LaunchDaemons/...vmnetd.plist # Prevented it from auto-starting
crc start # โœ… CRC now boots and works properly!

Let's approach this clearly, methodically, and comprehensively, as you've hit a very stubborn and persistent issue.

๐Ÿ”ด Your Current Issue (Recap):
The error you're seeing:

lua
Copy
Edit
Temporary error: pull secret not updated to disk
...
failed to read /var/lib/kubelet/config.json file: ssh command error:
err : Process exited with status 1
: /bin/bash: Input/output error
Means clearly:

CRC VM starts and runs โœ…

SSH succeeds initially โœ…

CRC repeatedly tries and fails to read or write the pull secret on disk โŒ

Eventually fails with a low-level /bin/bash: Input/output error โŒ

๐Ÿง  Potential Root Causes (What we've learned so far):

Possible Cause Likelihood (Your case) Explanation
Corrupted CRC Disk Image โš ๏ธ Very Likely Frequent cause after repeated CRC restarts or host interruption
Docker Desktop interference โš ๏ธ Very Likely vmnetd can corrupt vfkit networking (confirmed earlier)
Corporate VPN/Firewall โš ๏ธ Likely Blocks 127.0.0.1 or internal traffic silently
Insufficient RAM / CPU โš ๏ธ Possible Less likely, but can cause silent failures
macOS permissions โš ๏ธ Possible Disk permissions or macOS system policy issues
๐Ÿšจ Let's Do Comprehensive, Step-by-Step Troubleshooting:
Run these exact steps to pinpoint the issue.

โœ… STEP 1: Fully reset CRC again (with extra steps)
bash
Copy
Edit
crc stop
crc delete --force
crc cleanup

rm -rf ~/.crc
rm -rf ~/Library/Application\ Support/crc
rm -rf ~/.kube
rm -rf ~/.config/crc
โœ… STEP 2: Check and completely disable Docker Desktop Networking
bash
Copy
Edit
ps aux | grep vmnetd
If you see Docker again, remove it completely for now:

bash
Copy
Edit
sudo launchctl bootout system /Library/LaunchDaemons/com.docker.vmnetd.plist
sudo rm -f /Library/LaunchDaemons/com.docker.vmnetd.plist
Also quit Docker Desktop fully.

โœ… STEP 3: Disconnect VPN or Corporate Network (Critical Test)
Switch temporarily to your personal Wi-Fi hotspot (phone), just for this test.

Confirm no corporate firewall/VPN is active.

โœ… STEP 4: Reboot Your Mac (Absolutely critical step)
This ensures a fully clean vfkit/hypervisor environment.

โœ… STEP 5: Re-run CRC Setup Completely Fresh
bash
Copy
Edit
crc setup
crc start --memory 12288 --cpus 4 --pull-secret ~/Downloads/pull-secret.txt
(12GB RAM, 4 CPUs for stable startup)

โœ… STEP 6: If CRC still fails again, immediately do this:
Check the internal status inside CRC VM:

bash
Copy
Edit
crc ssh
Inside VM:

Check kubelet service:
bash
Copy
Edit
sudo systemctl status kubelet
sudo journalctl -xe -u kubelet
Check kubelet directory and file presence:
bash
Copy
Edit
ls -la /var/lib/kubelet/config.json
sudo cat /var/lib/kubelet/config.json
Check disk issues inside VM (Critical):
bash
Copy
Edit
dmesg | grep -i error
sudo journalctl -xe | grep -i error
๐Ÿงช STEP 7: Network Checks inside CRC VM (critical test):
bash
Copy
Edit
ping -c4 google.com
curl https://mirror.openshift.com
cat /etc/resolv.conf
ip route
๐Ÿ“Œ Interpreting Results (most likely outcomes):

Test If passes If fails
โœ… Ping Google (VM network test) Networking fine โœ… VPN/Firewall blocks VM access โŒ
โœ… curl mirror.openshift.com External DNS fine โœ… DNS broken or network blocked by firewall โŒ
โœ… kubelet status CRC VM internal services OK โœ… VM internal service failed due to config error โŒ
โœ… ls /var/lib/kubelet/config.json File present โœ… File missing or corrupted, CRC bootstrap failed โŒ
โœ… dmesg grep error Clean/no disk errors โœ…
โœ… Final Diagnosis (Your next action depends on these results):
If disk errors or file missing: CRC VM image corrupted โ†’ full reinstall (CRC + reboot)

If ping/curl fail: VPN/firewall โ†’ retry on personal hotspot

If kubelet/journalctl show errors: CRC internal issue (likely image or CRC bug)

๐Ÿ› ๏ธ Recommended Final Solutions (pick based on result):
โœ… Fully remove CRC and Docker Desktop, reboot, reinstall CRC completely fresh.

โœ… Run CRC only when disconnected from any corporate VPN or firewall.

โœ… Try CRC with slightly older stable bundle (like 4.13.x if 4.18.x fails repeatedly on your system).