Have you tried taking the metwork config out of the compose file and just letting podman handle it?

Just to offer the other perspective. I started with podman years ago. I knew very little about containers and I would say it made the learbing curve a lot steeper. Most guides and README’s use docker and when things didnt work I had to figure out if it was networking, selinux, rootless, not having the docker daemon, etc… without understanding fully what those things were because I didn’t know docker. But when I started running stuff on kubernetes, it was really easy. Pods in podman are isomorphic to kubernetes pods. I think the pain was worth it, but it was definitely not easy at the time. Documentation, guides, and networking have improved since then, so it may not be as big of a deal now

Quadlets with podman have completely replaced compose files for me. I use the kuberentes configs. Then I run a tailscale container in the pod and BAM, all of my computers can access that service without have to expose any ports.

Then I have an ansible playbook to log in to the host and start a detached tmux session so my user systemd services keep running. Its all rootless, and just so dang easy.