Node Pool Autoscaler Not Scaling Up Properly

Description:

The Cluster Autoscaler in GKE, which is responsible for adding or removing nodes based on resource demand, may not behave as expected. It can fail to add new nodes even when there is resource pressure.

Symptoms:

Pods remain pending despite node pool autoscaler being enabled.
Logs indicate that the autoscaler is trying to scale up, but no new nodes are provisioned.

Step 1: Verify Autoscaler Configuration

Ensure the autoscaler is enabled for the node pool:
gcloud container clusters update <cluster-name> --enable-autoscaling --min-nodes 1 --max-nodes 5 --node-pool <pool-name>

Step 2: Check Pod Resource Requests

Ensure that the pending pods have resource requests defined. Autoscaler only triggers if it detects resource pressure.

Step 3: Increase Node Pool Limits

If the max-nodes limit is reached, increase the limit:

gcloud container clusters update <cluster-name> --node-pool <pool-name> --max-nodes 10

Step 4: Check Quota Limits and IAM Permissions

Ensure your project has sufficient resource quotas to provision new nodes:

gcloud compute project-info describe --project <project-id>

Verify that the IAM roles for GKE service accounts allow autoscaler to manage the cluster resources.