By default, GCP Container Optimized (COS) VMs can't pull images on your Private Container Registry even if you've passed the right IAM and API credentials.
You'll get the below error message.
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
This baffled me. I had a working private container image specified during VM creation (us.gcr.io).
Turns out you need to add this single line to the startup-script. This gives docker the ability to pull from your private registry. Not sure why this was hidden in Google's Docs.
docker-credential-gcr configure-docker