AWX Tips & Tricks on CloudSOE Images

A collection of practical advice for running the CloudSOE AWX image in production, on either cloud.

Back up the database

Everything that matters in AWX — inventories, credentials, job history — lives in the PostgreSQL pod, whose data sits on the instance at /mnt/awx-storage/postgresql. Two complementary approaches:

Cloud snapshots (EBS snapshots on AWS, disk snapshots on Azure) capture the whole instance including the k3s state — the simplest full-recovery option.
Logical dumps restore across instances:

PG_POD=$(kubectl get pods | grep awx-postgres | cut -d " " -f1)
kubectl exec -i "$PG_POD" -- pg_dump -U awx awx | gzip > awx-db-$(date +%F).sql.gz

Ship dumps off the instance (S3, Azure Blob, anywhere).

Troubleshooting first boot

The first boot takes about eight minutes. If the progress bar seems stuck:

systemctl status awx-firstboot
journalctl -u awx-firstboot --no-pager
kubectl get pods        # are awx-web, awx-task and awx-postgres Running?

The bootstrap is safe to re-run: its completion marker (/var/lib/awx-firstboot.done) is only written on success, so a failed run retries on the next boot. A manual retry is just:

sudo systemctl restart awx-firstboot

The longest phase is the database migrations — the progress bar shows a live count of migrations remaining, so movement there is normal, not a hang.

Right-size the instance

AWX on k3s runs a control plane, web and task pods, PostgreSQL and Redis on one node. 8 GB RAM is a realistic floor; if jobs queue slowly or pods get OOM-killed (kubectl get events will show it), step up a size (t3.large → t3.xlarge on AWS, D2as_v4 → D4as_v4 on Azure). Concurrent job capacity scales with memory.

Keep the versions pinned in mind

The image pins AWX 24 and a matching AWX Operator version — a combination that is tested together. Upgrading AWX in place means upgrading the operator and the custom resource in lockstep; for major moves it’s safer to launch a fresh CloudSOE image and restore a database dump than to upgrade in place.

OS security updates are separate and safe: sudo apt update && sudo apt upgrade on your own schedule.

Watching jobs from the shell

The portal is the main interface, but the on-box kubectl is handy for a quick pulse check:

kubectl get pods                 # job pods appear alongside the core stack
TASK_POD=$(kubectl get pods | grep awx-task | cut -d " " -f1)
kubectl logs -f "$TASK_POD"      # live task-dispatcher logs

The MOTD shows the wrong IP

The login banner captures the public IP at first boot. If you later attach a new Elastic IP / public IP, the banner will be stale (AWX itself is unaffected — it serves on all interfaces). Edit /etc/motd to update it. On Azure, use a static public IP from the start to avoid the address changing across stop/start.

Verify the portal from outside

On Azure especially, a VM often cannot reach its own public IP from within. Always verify from your own machine:

curl -sI http://<public-ip>:30300/

A 200 (or a redirect to the login page) means AWX is up — see the configuration guide for putting a TLS-terminating load balancer in front.