awxbackups fail with => The task includes an option with an undefined variable

[salanisor]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Backups were working fine and suddenly started receiving the following error, similar to issue 1577.

Started with v2.17.0 & upgraded to see if the issue would go away.

awx-operator.v2.18.0                AWX                              2.18.0    awx-operator.v2.17.0

AWX Operator version

2.18.0

AWX version

v1beta1

Kubernetes platform

openshift

Kubernetes/Platform version

4.13.17

Modifications

no

Steps to reproduce

apply yaml

---
apiVersion: awx.ansible.com/v1beta1
kind: AWXBackup
metadata:
  name: awxbackup-test-sandbox-2
  namespace: awx
spec:
  deployment_name: awx-infra
  backup_pvc: awx-sandbox-backup-claim
  no_log: false

Expected results

successful backups - can actually see that the task returns the right field when testing. Should have probably used the same ansible version.

cat awx.json | jq '.this_awx.resources[0].status.postgresConfigurationSecret'
"awx-sandbox-postgres-configuration"

ansible-playbook --version
ansible-playbook [core 2.17.0]
  config file = None
  configured module search path = ['/Users/x/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/10.0.1/libexec/lib/python3.12/site-packages/ansible
  ansible collection location = /Users/x/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible-playbook
  python version = 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 15.0.0 (clang-1500.3.9.4)] (/opt/homebrew/Cellar/ansible/10.0.1/libexec/bin/python)
  jinja version = 3.1.4
  libyaml = True

Actual results

fails with error

TASK [backup :Get PostgreSQL configuration] ***********************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/backup/tasks/postgres.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED!
 
 => {\"msg\": \"The task includes an option with an undefined variable. The error was: list object has no element 0. list object has no element 0\\n\\nThe error appears to be in '/opt/ansible/roles/backup/tasks/postgres.yml'
 : line 3, column 3, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n- name: Get PostgreSQL configuration\\n  ^ here\\n\"}\u001b[0m\n\r\nPLAY RECAP 
 *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=16  \u001b[0m \u001b[0;33mchanged=2   \u001b[0m unreachable=0    \u001b[0;31mfailed=1 
 \u001b[0m \u001b[0;36mskipped=8   \u001b[0m rescued=0    ignored=0   \n","job":"8280544932261614561","name":"awxbackup-test-sandbox-2","namespace":"awx","error":"exit status 2","stacktrace":"github.com/operator-framework
 /ansible-operator-plugins/internal/ansible/runner.(*runner).Run.func1\n\tansible-operator-plugins/internal/ansible/runner/runner.go:269"}

Additional information

awx definition, this was working fine and had run a few recovery tests in this namespace.

We had installed OCP service mesh and thought it may be the culprit but no. Removed it and issue persists.

 oc get awxbackups
NAME                   AGE
awxbackup-2024         43d
awxbackup-2024-06-03   14d

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    argocd.argoproj.io/sync-wave: "200"
  name: awx-sandbox
  namespace: awx
spec:
  postgres_keepalives_count: 5
  postgres_keepalives_idle: 5
  ee_resource_requirements:
    requests:
      cpu: 50m
      memory: 64M
  create_preload_data: true
  garbage_collect_secrets: false
  loadbalancer_port: 80
  no_log: true
  task_resource_requirements:
    requests:
      cpu: 50m
      memory: 128M
  image_pull_policy: IfNotPresent
  loadbalancer_ip: ''
  projects_storage_size: 8Gi
  auto_upgrade: true
  task_privileged: false
  postgres_keepalives: true
  postgres_keepalives_interval: 5
  ipv6_disabled: false
  projects_storage_access_mode: ReadWriteMany
  set_self_labels: true
  web_resource_requirements:
    requests:
      cpu: 50m
      memory: 128M
  projects_persistence: false
  replicas: 1
  admin_user: admin
  loadbalancer_protocol: http
  
  # Set the default admin password secret 
  admin_password_secret: awx-admin-secret
  
  # Give route information for this instance 
  service_type: ClusterIP
  ingress_type: route
  route_host: awx-sandbox.apps.help.example.com
  route_tls_termination_mechanism: Edge
  hostname: awx-sandbox.apps.help.example.com
  
  # Configure this instance to trust OpenLDAP for access checks and for general lookups 
  ldap_cacert_secret: awx-ca-secret
  bundle_cacert_secret: awx-ca-secret
  
  # Configure this instance to successfully perform Kerberos lookups in domain
  # First, create the ConfigMap as part of the kustomization.yaml 
  # Second, turn the ConfigMap into a volume 
  # Third, mount the volume to the Pod Members 
  extra_volumes: |
    - name: awx-corp-krb5
      configMap:
        defaultMode: 420
        items:
        - key: krb5.conf
          path: krb5.conf
        name: awx-corp-krb5-conf-configmap
  web_extra_volume_mounts: |
    - name: awx-corp-krb5
      mountPath: /etc/krb5.conf
      subPath: krb5.conf
  task_extra_volume_mounts: |
    - name: awx-corp-krb5
      mountPath: /etc/krb5.conf
      subPath: krb5.conf
  ee_extra_volume_mounts: |
    - name: awx-corp-krb5
      mountPath: /etc/krb5.conf
      subPath: krb5.conf

Operator Logs

log.tar.gz

[D1StrX]

Yup, same as in here: #1518

[djyasin]

Hello @salanisor, is this the same issue described in #1518?