Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS Sequoia(15.3.1) with M4 Pro chip EOF on podman machine start #25402

Open
Coosis opened this issue Feb 26, 2025 · 4 comments
Open

macOS Sequoia(15.3.1) with M4 Pro chip EOF on podman machine start #25402

Coosis opened this issue Feb 26, 2025 · 4 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. machine macos MacOS (OSX) related

Comments

@Coosis
Copy link

Coosis commented Feb 26, 2025

Issue Description

podman machine start results in:

Error: Get "http://localhost:{some port}/vm/state": EOF

Steps to reproduce the issue

Steps to reproduce the issue

$ sudo rm -rf /opt/podman
$ rm -rf ~/.config/containers ~/.local/share/containers
$ brew uninstall podman
$ brew uninstall podman-desktop
$ brew uninstall qemu

$ brew install podman
$ podman machine init mini-podman-machine --cpus 8 --disk-size 200 -m 12288
Looking up Podman Machine image at quay.io/podman/machine-os:5.4 to create VM
Getting image source signatures
Copying blob a5202b5e26ab done   |
Copying config 44136fa355 done   |
Writing manifest to image destination
a5202b5e26abdb846baed95761c47a42c5d49529752fa6fba45ea7a5ea046b52
Extracting compressed file: mini-podman-machine-arm64.raw: done
Machine init complete
To start your machine run:

        podman machine start mini-podman-machine

$ podman machine start mini-podman-machine
Starting machine "mini-podman-machine"
Error: Get "http://localhost:51892/vm/state": EOF

Describe the results you received

$ podman machine start mini-podman-machine --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `applehv` virtualization provider
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv/mini-podman-machine.json is 78
Starting machine "mini-podman-machine"
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv/mini-podman-machine.json is 78
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/libkrun is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/libkrun is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/libkrun/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
Error: Get "http://localhost:51892/vm/state": EOF
DEBU[0000] Shutting down engines

cat /Users/vlad/.config/containers/podman/machine/applehv/mini-podman-machine.json | jq:

{
  "Created": "2025-02-26T13:42:58.2599+08:00",
  "GvProxy": {
    "Debug": false,
    "MTU": 0,
    "LogFile": "",
    "PidFile": "",
    "SSHPort": 0
  },
  "HostUser": {
    "Rootful": false,
    "UID": 501,
    "HostUserModified": false
  },
  "LastUp": "0001-01-01T00:00:00Z",
  "Mounts": [
    {
      "OriginalInput": "",
      "ReadOnly": false,
      "Source": "/Users",
      "Tag": "a2a0ee2c717462feb1de2f5afd59de5fd2d8",
      "Target": "/Users",
      "Type": "virtiofs",
      "VSockNumber": null
    },
    {
      "OriginalInput": "",
      "ReadOnly": false,
      "Source": "/private",
      "Tag": "71708eb255bc230cd7c91dd26f7667a7b938",
      "Target": "/private",
      "Type": "virtiofs",
      "VSockNumber": null
    },
    {
      "OriginalInput": "",
      "ReadOnly": false,
      "Source": "/var/folders",
      "Tag": "a0bb3a2c8b0b02ba5958b0576f0d6530e104",
      "Target": "/var/folders",
      "Type": "virtiofs",
      "VSockNumber": null
    }
  ],
  "Name": "mini-podman-machine",
  "Resources": {
    "CPUs": 8,
    "DiskSize": 200,
    "Memory": 12288,
    "USBs": []
  },
  "SSH": {
    "IdentityPath": "/Users/vlad/.local/share/containers/podman/machine/machine",
    "Port": 51797,
    "RemoteUsername": "core"
  },
  "Version": 1,
  "ImagePath": {
    "Path": "/Users/vlad/.local/share/containers/podman/machine/applehv/mini-podman-machine-arm64.raw"
  },
  "AppleHypervisor": {
    "Vfkit": {
      "LogLevel": "panic",
      "Endpoint": "http://localhost:51892",
      "BinaryPath": null,
      "VirtualMachine": {
        "vcpus": 8,
        "memoryBytes": 12884901888,
        "bootloader": {
          "kind": "efiBootloader",
          "efiVariableStorePath": "/Users/vlad/.local/share/containers/podman/machine/applehv/efi-bl-mini-podman-machine",
          "createVariableStore": true
        }
      },
      "Rosetta": true
    }
  },
  "Starting": false,
  "Rosetta": false,
  "Ansible": null
}

Describe the results you expected

Podman starts machine successfully

podman info output

$ podman info
OS: darwin/arm64
buildOrigin: brew
provider: applehv
version: 5.4.0

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:51797: connect: connection refused

$ podman system connection list
Name                      URI                                                         Identity                                                    Default     ReadWrite
mini-podman-machine       ssh://core@127.0.0.1:51797/run/user/501/podman/podman.sock  /Users/vlad/.local/share/containers/podman/machine/machine  true        true
mini-podman-machine-root  ssh://root@127.0.0.1:51797/run/podman/podman.sock           /Users/vlad/.local/share/containers/podman/machine/machine  false       true

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

$ podman version
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:51797: connect: connection refused

$ podman machine ls
Error: Get "http://localhost:51892/vm/state": EOF

$ podman machine reset
Error: Get "http://localhost:53295/vm/state": EOF

$ podman system connection list
Name                      URI                                                         Identity                                                    Default     ReadWrite
mini-podman-machine       ssh://core@127.0.0.1:51797/run/user/501/podman/podman.sock  /Users/vlad/.local/share/containers/podman/machine/machine  true        true
mini-podman-machine-root  ssh://root@127.0.0.1:51797/run/podman/podman.sock           /Users/vlad/.local/share/containers/podman/machine/machine  false       true

$ which podman
/opt/homebrew/bin/podman

$ which vfkit
/opt/homebrew/bin/vfkit

Additional information

I've tried solutions from #25015, as well as some others:

  1. Install podman-desktop and use it to install podman
  2. Install podman directly from release page & pkg installer
  3. Setting env to applehv before creating machine
@Coosis Coosis added the kind/bug Categorizes issue or PR as related to a bug. label Feb 26, 2025
@Luap99 Luap99 added macos MacOS (OSX) related machine labels Feb 26, 2025
@baude
Copy link
Member

baude commented Feb 26, 2025

to me, this looks like gvproxy is failing to respond. oddly, i also dont see the debug message where it was started. Could you consider the following?

  • remove the brew install and reinstall with just the podman release
  • run everything with --log-level debug. the start debug provided seems like it is incomplete.
  • with the debug log level, do you see a GUI pop up and in that GUI do you see the operating system boot ?

@Coosis
Copy link
Author

Coosis commented Feb 26, 2025

to me, this looks like gvproxy is failing to respond. oddly, i also dont see the debug message where it was started. Could you consider the following?

  • remove the brew install and reinstall with just the podman release
  • run everything with --log-level debug. the start debug provided seems like it is incomplete.
  • with the debug log level, do you see a GUI pop up and in that GUI do you see the operating system boot ?

Thank you for responding! Here(After installing release 5.4.0 from release):
podman machine init part:

$ which podman
/opt/podman/bin/podman

$ podman machine init --log-level debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `applehv` virtualization provider
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv/podman-machine-default.json is 81
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/podman-machine-default-arm64.raw is 91
Looking up Podman Machine image at quay.io/podman/machine-os:5.4 to create VM
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying to access "quay.io/podman/machine-os:5.4"
DEBU[0000] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.config/containers/auth.json
DEBU[0000] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.config/containers/auth.json
DEBU[0000] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.docker/config.json
DEBU[0000] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.dockercfg
DEBU[0000] No credentials for quay.io/podman/machine-os found
DEBU[0000]  No signature storage configuration found for quay.io/podman/machine-os:5.4, using built-in default file:///Users/vlad/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io
DEBU[0000] GET https://quay.io/v2/
DEBU[0000] Ping https://quay.io/v2/ status 401
DEBU[0000] GET https://quay.io/v2/auth?scope=repository%3Apodman%2Fmachine-os%3Apull&service=quay.io
DEBU[0001] Increasing token expiration to: 60 seconds
DEBU[0001] GET https://quay.io/v2/podman/machine-os/manifests/5.4
DEBU[0002] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json"
DEBU[0002] found image in digest: "sha256:81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2"
DEBU[0002] GET https://quay.io/v2/podman/machine-os/manifests/sha256:81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2
DEBU[0003] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0003] original artifact file name: podman-machine.aarch64.applehv.raw.zst
DEBU[0003] GET https://quay.io/v2/podman/machine-os/manifests/sha256:81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2
DEBU[0003] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0003] original artifact file name: podman-machine.aarch64.applehv.raw.zst
DEBU[0003] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache/81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2.raw.zst is 137
DEBU[0003] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache/81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2 is 129
DEBU[0003] no signature policy file found: using default allow everything signature policy
DEBU[0003] Using registries.d directory /etc/containers/registries.d
DEBU[0003] Trying to access "quay.io/podman/machine-os@sha256:81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2"
DEBU[0003] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.config/containers/auth.json
DEBU[0003] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.config/containers/auth.json
DEBU[0003] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.docker/config.json
DEBU[0003] No credentials matching quay.io/podman/machine-os found in /Users/vlad/.dockercfg
DEBU[0003] No credentials for quay.io/podman/machine-os found
DEBU[0003]  No signature storage configuration found for quay.io/podman/machine-os@sha256:81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2, using built-in default file:///Users/vlad/.local/share/containers/sigstore
DEBU[0003] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io
DEBU[0003] GET https://quay.io/v2/
DEBU[0004] Ping https://quay.io/v2/ status 401
DEBU[0004] GET https://quay.io/v2/auth?scope=repository%3Apodman%2Fmachine-os%3Apull&service=quay.io
DEBU[0005] Increasing token expiration to: 60 seconds
DEBU[0005] GET https://quay.io/v2/podman/machine-os/manifests/sha256:81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2
DEBU[0005] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0005] Error creating a SQLite blob info cache at /Users/vlad/.local/share/containers/cache/blob-info-cache-v1.sqlite, using a memory-only cache: beginning transaction: Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub
DEBU[0005] IsRunningImageAllowed for image docker:quay.io/podman/machine-os@sha256:81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2
DEBU[0005]  Using default policy section
DEBU[0005]  Requirement 0: allowed
DEBU[0005] Overall: allowed
Getting image source signatures
DEBU[0005] Reading /Users/vlad/.local/share/containers/sigstore/podman/machine-os@sha256=81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2/signature-1
DEBU[0005] Not looking for sigstore attachments: disabled by configuration
DEBU[0005] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json]
DEBU[0005] ... will first try using the original manifest unmodified
DEBU[0005] Checking if we can reuse blob sha256:a5202b5e26abdb846baed95761c47a42c5d49529752fa6fba45ea7a5ea046b52: general substitution = true, compression for MIME type "application/zstd" = false
DEBU[0005] Downloading /v2/podman/machine-os/blobs/sha256:a5202b5e26abdb846baed95761c47a42c5d49529752fa6fba45ea7a5ea046b52
DEBU[0005] GET https://quay.io/v2/podman/machine-os/blobs/sha256:a5202b5e26abdb846baed95761c47a42c5d49529752fa6fba45ea7a5ea046b52
Copying blob a5202b5e26ab [--------------------------------------] 0.0b / 884.8MiB | 0.0 b/s
DEBU[0007] Detected compression format zstd
DEBU[0007] Compression change for blob sha256:a5202b5e26abdb846baed95761c47a42c5d49529752fa6fba45ea7a5ea046b52 ("application/zstd") not supported
Copying blob a5202b5e26ab done   |
DEBU[0089] Downloading /v2/podman/machine-os/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
DEBU[0089] GET https://quay.io/v2/auth?scope=repository%3Apodman%2Fmachine-os%3Apull&service=quay.io
Copying config 44136fa355 [--------------------------------------] 0.0b / 2.0b | 0.0 b/s
DEBU[0089] Increasing token expiration to: 60 seconds
Copying config 44136fa355 [--------------------------------------] 0.0b / 2.0b | 0.0 b/s
Copying config 44136fa355 done   |
DEBU[0090] Compression change for blob sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a ("application/vnd.oci.empty.v1+json") not supported
Copying config 44136fa355 done   |
Writing manifest to image destination
DEBU[0090] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache/81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2 is 129
DEBU[0090] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache/81ba5453febd6cba048ef0101563b4b1cd087052c42a43690abdfef8f7569db2.raw.zst is 137
a5202b5e26abdb846baed95761c47a42c5d49529752fa6fba45ea7a5ea046b52
DEBU[0090] Detected compression format zstd
Extracting compressed file: podman-machine-default-arm64.raw: done
DEBU[0096] --> imagePath is "/Users/vlad/.local/share/containers/podman/machine/applehv/podman-machine-default-arm64.raw"
DEBU[0096] socket length for /Users/vlad/.config/containers/podman/machine/applehv/podman-machine-default.ign is 80
DEBU[0096] resizing /Users/vlad/.local/share/containers/podman/machine/applehv/podman-machine-default-arm64.raw to 107374182400 bytes
DEBU[0096] writing ignition file to "/Users/vlad/.config/containers/podman/machine/applehv/podman-machine-default.ign"
DEBU[0096] writing configuration file "/Users/vlad/.config/containers/podman/machine/applehv/podman-machine-default.json"
Machine init complete
To start your machine run:

        podman machine start

DEBU[0096] Called machine init.PersistentPostRunE(podman machine init --log-level debug)
DEBU[0096] Shutting down engines

podman machine start part:

$ podman machine start --log-level debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `applehv` virtualization provider
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv/podman-machine-default.json is 81
Starting machine "podman-machine-default"
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/applehv/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/applehv/podman-machine-default.json is 81
DEBU[0000] socket length for /Users/vlad/.config/containers/podman/machine/libkrun is 53
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/libkrun is 58
DEBU[0000] socket length for /Users/vlad/.local/share/containers/podman/machine/libkrun/cache is 64
DEBU[0000] socket length for /var/folders/hp/mlps7qpj01ncw46lqk1ljq7r0000gn/T/podman is 55
Error: Get "http://localhost:61696/vm/state": EOF
DEBU[0000] Shutting down engines

No, i do not see any kind of gui popping up.
Just in case, here's a screenshot just to prove this is indeed all the output:

Image

@baude
Copy link
Member

baude commented Feb 26, 2025

while that start is running ... can you ps aux | grep vfkit and same for gvproxy ? I'm thinking both will not be running ?

@Coosis
Copy link
Author

Coosis commented Feb 26, 2025

while that start is running ... can you ps aux | grep vfkit and same for gvproxy ? I'm thinking both will not be running ?

that start only takes ~0.04 seconds on my machine before failing... Is there a way to execute ps aux | grep vfkit within that time period?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. machine macos MacOS (OSX) related
Projects
None yet
Development

No branches or pull requests

3 participants