Skip to content

Troubleshooting Guide

This guide covers common issues and solutions encountered during development and deployment.

Cloud Run Deployment Issues

Container Failed to Start

Error: The user-provided container failed to start and listen on the port defined provided by the PORT=8000 environment variable

Common Causes: 1. Database connection failures during startup 2. Missing environment variables 3. Port configuration issues 4. Cloud SQL connection problems

Solutions: 1. Check Cloud SQL Connection: - Ensure Cloud Run service has the proper annotation: run.googleapis.com/cloudsql-instances - Verify DATABASE_URL format: postgresql://user:pass@/dbname?host=/cloudsql/project:region:instance - Check that the service account has roles/cloudsql.client permission

  1. Verify Environment Variables:
  2. Check that all required secrets are populated in Secret Manager
  3. Verify the service account has roles/secretmanager.secretAccessor permission

  4. Test with: gcloud secrets versions access latest --secret="secret-name"

  5. Database Connection Format: 

    # Correct format for Cloud SQL
DATABASE_URL="postgresql://user:pass@/dbname?host=/cloudsql/project:region:instance"

# Incorrect format (will fail)
DATABASE_URL="postgresql://user:pass@private-ip:5432/dbname"

AI Service Initialization Errors

Error: No valid AI provider configuration found

Solution: 1. Check that ANTHROPIC_API_KEY and GOOGLE_AI_API_KEY are set in Secret Manager 2. Verify the AI_PROVIDER environment variable is set correctly 3. Test API keys manually

Secret Manager Access Issues

Error: Permission denied when accessing secrets

Solution: 1. Verify the Cloud Run service account has the correct IAM roles: 

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:service-account@project.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

  1. Check that secrets exist and have versions: 
    gcloud secrets list --project=PROJECT_ID
gcloud secrets versions list SECRET_NAME --project=PROJECT_ID

Database Issues

PostgreSQL Connection Errors

Error: connection to server on socket "/cloudsql/..." failed: No such file or directory

Causes: 1. Cloud SQL proxy not configured 2. Incorrect connection string format 3. Service account lacks Cloud SQL permissions

Solutions: 1. Add Cloud SQL annotation to Cloud Run service 2. Use correct connection string format 3. Grant roles/cloudsql.client to service account

Cache Connection Issues

Error: Cache entries table missing

Solution: 1. Ensure the CacheEntry model is imported in app/models/__init__.py 2. Run database migrations or table creation 3. Verify cache type is set to postgres in environment variables

Frontend Issues

CORS Errors

Error: Access to XMLHttpRequest blocked by CORS policy

Solution: 1. Add frontend domain to BACKEND_CORS_ORIGINS_STR environment variable 2. Ensure the backend is deployed with updated CORS settings 3. Test with curl to verify CORS headers are present

API Connection Errors

Error: Network Error when calling API

Causes: 1. Backend service not running 2. Incorrect API endpoint URL 3. Backend returning 500 errors

Solutions: 1. Check backend health: curl https://api-url/health 2. Verify API endpoint configuration 3. Check backend logs for errors

Infrastructure Issues

Terraform Cloud Deployment

Error: Terraform plan/apply failures

Solutions: 1. Verify all required environment variables are set in Terraform Cloud 2. Check that the workspace is connected to the correct VCS branch 3. Ensure proper IAM permissions for the service account

Secret Manager Setup

Error: Secrets not found or empty

Solution: 1. Create secrets with proper naming convention: secret-name-environment 2. Add secret versions with actual values 3. Grant access to the appropriate service accounts

Debugging Commands

Cloud Run Logs

# Get recent logs
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=SERVICE_NAME" --limit=50 --project=PROJECT_ID

# Get error logs only
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=SERVICE_NAME AND severity>=ERROR" --limit=20 --project=PROJECT_ID

Secret Manager

# List secrets
gcloud secrets list --project=PROJECT_ID

# Get secret value
gcloud secrets versions access latest --secret="secret-name" --project=PROJECT_ID

# Add secret version
gcloud secrets versions add "secret-name" --data-file=<(echo "secret-value") --project=PROJECT_ID

Cloud SQL

# Test connection
gcloud sql connect INSTANCE_NAME --user=postgres --project=PROJECT_ID

# Check instance status
gcloud sql instances describe INSTANCE_NAME --project=PROJECT_ID

Service Health Checks

# Backend health
curl https://api-url/health
curl https://api-url/service/health

# Frontend health
curl https://frontend-url/

# Documentation
curl https://docs.macumbatravel.com/

Performance Monitoring

Cloud Run Metrics

  • Monitor request latency and error rates in Cloud Console
  • Check memory and CPU usage
  • Review startup time metrics

Database Performance

  • Monitor connection pool usage
  • Check query performance
  • Review connection counts

Common Environment Variables

Backend Required Variables

# Core
APP_ENV=development
DATABASE_URL=postgresql://user:pass@/db?host=/cloudsql/project:region:instance
CACHE_TYPE=postgres

# AI Services
AI_PROVIDER=gemini
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_AI_API_KEY=AIza...

# External APIs
GOOGLE_MAPS_API_KEY=AIza...
SENDGRID_API_KEY=SG....
RAPIDAPI_KEY=abc...

# CORS
BACKEND_CORS_ORIGINS_STR=http://localhost:5173,https://dev.macumbatravel.com

Frontend Required Variables

# API Configuration
VITE_API_URL=https://api-dev.macumbatravel.com
VITE_APP_ENV=development

Support

For additional support: - Check the Architecture Guide - Review Deployment Documentation - Consult Database Guide - Examine recent commit history for related fixes