2026-07-29T07:51:31.944249379Z [disk usage checker] Started
2026-07-29T07:51:31.947810539Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-29T07:51:32.147614499Z Gathering data for ns/openshift-cluster-version...
2026-07-29T07:51:32.294141574Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T07:51:32.389024359Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T07:51:32.473691009Z Gathering data for ns/default...
2026-07-29T07:51:32.781254979Z Gathering data for ns/openshift...
2026-07-29T07:51:33.200664264Z Gathering data for ns/kube-system...
2026-07-29T07:51:33.883661454Z Gathering data for ns/openshift-etcd...
2026-07-29T07:51:34.194515428Z Wrote inspect data to must-gather.
2026-07-29T07:51:34.194548940Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T07:51:34.194548940Z     namespaces "assisted-installer" not found
2026-07-29T07:51:34.550900674Z Waiting on subprocesses to finish execution.
2026-07-29T07:51:34.556209735Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-29T07:51:34.564401188Z Executing Istio gather script
2026-07-29T07:51:34.565731751Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-29T07:51:34.576985613Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-29T07:51:34.577587842Z INFO: Collecting host service logs for crio
2026-07-29T07:51:34.578004531Z INFO: Collecting host service logs for kubelet
2026-07-29T07:51:34.578325277Z INFO: Collecting host service logs for rpm-ostreed
2026-07-29T07:51:34.578633136Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-29T07:51:34.578969100Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-29T07:51:34.579273840Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-29T07:51:34.579644163Z INFO: Collecting host service logs for NetworkManager
2026-07-29T07:51:34.580055137Z INFO: Collecting host service logs for openvswitch
2026-07-29T07:51:34.580352339Z INFO: Collecting host service logs for ovs-configuration
2026-07-29T07:51:34.580643837Z INFO: Collecting host service logs for ovsdb-server
2026-07-29T07:51:34.580961366Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-29T07:51:34.581295100Z INFO: Waiting for worker host service log collection to complete ...
2026-07-29T07:51:34.586213055Z INFO: Gathering HAProxy config files
2026-07-29T07:51:34.911887336Z INFO: Waiting for node performance related collection to complete ...
2026-07-29T07:51:35.063465148Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-29T07:51:35.574993510Z ERROR: No running kube-apiserver pods found
2026-07-29T07:51:35.668760544Z No resources found in openshift-etcd namespace.
2026-07-29T07:51:35.857441880Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-29T07:51:35.919852034Z Wrote inspect data to must-gather.
2026-07-29T07:51:35.974719369Z INFO: "metallb-operator" not detected. Skipping.
2026-07-29T07:51:36.065296734Z error: the server doesn't have a resource type "performanceprofile"
2026-07-29T07:51:36.256335836Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-29T07:51:36.287804520Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-29T07:51:36.290153835Z INFO: Worker host service log collection to complete.
2026-07-29T07:51:36.320709136Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-x8tsg
2026-07-29T07:51:36.320709136Z insights-runtime-extractor-8qj6f
2026-07-29T07:51:36.320709136Z insights-runtime-extractor-f9457
2026-07-29T07:51:36.320709136Z insights-runtime-extractor-w7gfm
2026-07-29T07:51:36.366666810Z Wrote inspect data to must-gather.
2026-07-29T07:51:36.404510912Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-29T07:51:36.404593522Z INFO: on-disk MachineConfig config collection complete.
2026-07-29T07:51:36.417613464Z No resources found
2026-07-29T07:51:36.548076953Z No resources found
2026-07-29T07:51:36.686134357Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T07:51:36.732390372Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-29T07:51:36.794057118Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-29T07:51:36.817269337Z INFO: 'previous-logs' folder not found on ip-10-0-131-190.ec2.internal, skipping...
2026-07-29T07:51:36.864340691Z Wrote inspect data to must-gather.
2026-07-29T07:51:36.910082607Z Wrote inspect data to must-gather.
2026-07-29T07:51:36.930714384Z Wrote inspect data to must-gather.
2026-07-29T07:51:36.968408461Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-29T07:51:36.968408461Z See 'oc rsync -h' for help and examples
2026-07-29T07:51:36.986617396Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-29T07:51:37.009153762Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-29T07:51:37.022263057Z Wrote inspect data to must-gather.
2026-07-29T07:51:37.084299178Z Wrote inspect data to must-gather.
2026-07-29T07:51:37.264565471Z error: the server doesn't have a resource type "machineconfigs"
2026-07-29T07:51:37.298289690Z INFO: 'previous-logs' folder not found on ip-10-0-131-39.ec2.internal, skipping...
2026-07-29T07:51:37.301466109Z tar: Removing leading `/' from member names
2026-07-29T07:51:37.307125341Z INFO: HAProxy config collection complete.
2026-07-29T07:51:37.327182702Z Wrote inspect data to must-gather.
2026-07-29T07:51:37.422222710Z INFO: Getting rules from prometheus-k8s-0
2026-07-29T07:51:37.475733244Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-29T07:51:37.484370074Z Gathering data for ns/openshift-monitoring...
2026-07-29T07:51:37.641220129Z INFO: 'previous-logs' folder not found on ip-10-0-137-121.ec2.internal, skipping...
2026-07-29T07:51:37.641341580Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-29T07:51:37.641360642Z INFO: Machine Config Daemon termination log collection complete.
2026-07-29T07:51:37.679783748Z Wrote inspect data to must-gather.
2026-07-29T07:51:37.711542628Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-29T07:51:37.741176341Z Wrote inspect data to must-gather.
2026-07-29T07:51:37.883895213Z INFO: Getting status/config from prometheus-k8s-0
2026-07-29T07:51:37.908715934Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-29T07:51:37.923602483Z Wrote inspect data to must-gather.
2026-07-29T07:51:38.128487987Z Wrote inspect data to must-gather.
2026-07-29T07:51:38.220107305Z INFO: INTERCONNECT MODE
2026-07-29T07:51:38.220107305Z INFO: Gathering ovn-kubernetes DBs
2026-07-29T07:51:38.285900249Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-29T07:51:38.357719468Z Wrote inspect data to must-gather.
2026-07-29T07:51:38.379593388Z INFO: Gathering OVN_Northbound from ovnkube-node-qg44w...
2026-07-29T07:51:38.380158398Z INFO: Gathering OVN_Northbound from ovnkube-node-tf55x...
2026-07-29T07:51:38.380463104Z INFO: Gathering OVN_Northbound from ovnkube-node-x5bz9...
2026-07-29T07:51:38.380759648Z INFO: Gathering OVN_Southbound from ovnkube-node-qg44w...
2026-07-29T07:51:38.381048804Z INFO: Gathering OVN_Southbound from ovnkube-node-tf55x...
2026-07-29T07:51:38.381313521Z INFO: Gathering OVN_Southbound from ovnkube-node-x5bz9...
2026-07-29T07:51:38.388386079Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-29T07:51:38.580708789Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-29T07:51:38.690569866Z tar: Removing leading `/' from member names
2026-07-29T07:51:38.711139503Z Wrote inspect data to must-gather.
2026-07-29T07:51:38.718479907Z tar: Removing leading `/' from member names
2026-07-29T07:51:38.731824956Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:38.750242966Z tar: Removing leading `/' from member names
2026-07-29T07:51:38.752918812Z tar: Removing leading `/' from member names
2026-07-29T07:51:38.753057147Z tar: Removing leading `/' from member names
2026-07-29T07:51:38.807905886Z tar: Removing leading `/' from member names
2026-07-29T07:51:38.899447046Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-29T07:51:38.954447155Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T07:51:38.995810314Z Wrote inspect data to must-gather.
2026-07-29T07:51:39.014883478Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:39.019324964Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-29T07:51:39.183510841Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-29T07:51:39.248485838Z Wrote inspect data to must-gather.
2026-07-29T07:51:39.262305974Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:39.271434073Z INFO: Waiting for network log collection to complete ...
2026-07-29T07:51:39.271468281Z INFO: Waiting for ovnk database copies to complete ...
2026-07-29T07:51:39.271877847Z INFO: Copying ovnk databases complete.
2026-07-29T07:51:39.274117818Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-29T07:51:39.276816387Z ovnk_database_store/
2026-07-29T07:51:39.276891081Z ovnk_database_store/ovnkube-node-x5bz9_sbdb
2026-07-29T07:51:39.296133083Z ovnk_database_store/ovnkube-node-x5bz9_nbdb
2026-07-29T07:51:39.303913309Z ovnk_database_store/ovnkube-node-tf55x_sbdb
2026-07-29T07:51:39.323727984Z ovnk_database_store/ovnkube-node-qg44w_nbdb
2026-07-29T07:51:39.331273118Z ovnk_database_store/ovnkube-node-tf55x_nbdb
2026-07-29T07:51:39.338716517Z ovnk_database_store/ovnkube-node-qg44w_sbdb
2026-07-29T07:51:39.360208489Z INFO: Network log collection complete.
2026-07-29T07:51:39.481934737Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:39.482080617Z W0729 07:51:39.482051    1513 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-29T07:51:39.485349101Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-29T07:51:39.489492563Z Wrote inspect data to must-gather.
2026-07-29T07:51:39.558229741Z INFO: Getting status from alertmanager-main-0
2026-07-29T07:51:39.619457764Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T07:51:39.696573101Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:39.792017184Z Wrote inspect data to must-gather.
2026-07-29T07:51:39.810495380Z Gathering data for ns/openshift-network-console...
2026-07-29T07:51:39.905340414Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:39.909156815Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-29T07:51:40.072741559Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-29T07:51:40.075530530Z INFO: Fallback to identify the container image from release info
2026-07-29T07:51:40.148604198Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:40.255690279Z Gathering data for ns/openshift-console-operator...
2026-07-29T07:51:40.367703152Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:40.370581570Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-29T07:51:40.517382630Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:40.642193745Z Gathering data for ns/openshift-console...
2026-07-29T07:51:40.700271536Z Wrote inspect data to /must-gather/istio.
2026-07-29T07:51:40.829808466Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-29T07:51:40.943465440Z daemonset.apps/perf-node-gather-daemonset created
2026-07-29T07:51:41.088270703Z Waiting for performance profile collector pods to become ready: 1
2026-07-29T07:51:41.177007700Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-29T07:51:41.402533216Z Done executing Istio gather script
2026-07-29T07:51:41.686577463Z Gathering data for ns/openshift-dns-operator...
2026-07-29T07:51:41.959440108Z Gathering data for ns/openshift-dns...
2026-07-29T07:51:41.992723443Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-29T07:51:42.189215506Z Waiting for performance profile collector pods to become ready: 2
2026-07-29T07:51:42.572219002Z Gathering data for ns/openshift-image-registry...
2026-07-29T07:51:43.013169603Z Gathering data for ns/openshift-ingress-operator...
2026-07-29T07:51:43.312376845Z Waiting for performance profile collector pods to become ready: 3
2026-07-29T07:51:43.355516035Z Gathering data for ns/openshift-ingress...
2026-07-29T07:51:43.675369881Z Gathering data for ns/openshift-ingress-canary...
2026-07-29T07:51:44.057155764Z Gathering data for ns/openshift-insights...
2026-07-29T07:51:44.426122195Z Waiting for performance profile collector pods to become ready: 4
2026-07-29T07:51:45.535888338Z Waiting for performance profile collector pods to become ready: 5
2026-07-29T07:51:45.865227000Z Gathering data for ns/openshift-jobset-operator...
2026-07-29T07:51:46.627011857Z Gathering data for ns/openshift-config...
2026-07-29T07:51:46.646480629Z Waiting for performance profile collector pods to become ready: 6
2026-07-29T07:51:46.884864438Z Gathering data for ns/openshift-config-managed...
2026-07-29T07:51:46.997413279Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-29T07:51:47.294429579Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-29T07:51:47.564901511Z Gathering data for ns/openshift-kube-apiserver...
2026-07-29T07:51:47.761021022Z Waiting for performance profile collector pods to become ready: 7
2026-07-29T07:51:47.824277470Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-29T07:51:48.104575804Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-29T07:51:48.378698068Z Gathering data for ns/openshift-kube-scheduler...
2026-07-29T07:51:48.633225795Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-29T07:51:48.892766271Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-29T07:51:48.929456491Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-29T07:51:49.135748230Z Collecting performance related data for node ip-10-0-131-39.ec2.internal
2026-07-29T07:51:49.144432544Z Collecting performance related data for node ip-10-0-131-190.ec2.internal
2026-07-29T07:51:49.153970746Z Collecting performance related data for node ip-10-0-137-121.ec2.internal
2026-07-29T07:51:49.315334732Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-29T07:51:49.676392095Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-29T07:51:50.304796267Z Gathering data for ns/openshift-multus...
2026-07-29T07:51:51.802700727Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-29T07:51:52.003647817Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-29T07:51:52.999505843Z Gathering data for ns/openshift-host-network...
2026-07-29T07:51:53.354289741Z Gathering data for ns/openshift-network-diagnostics...
2026-07-29T07:51:53.883905947Z Gathering data for ns/openshift-network-node-identity...
2026-07-29T07:51:54.303913589Z Gathering data for ns/openshift-network-operator...
2026-07-29T07:51:54.681890598Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-29T07:51:54.952115748Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-29T07:51:55.440163927Z Gathering data for ns/openshift-apiserver-operator...
2026-07-29T07:51:55.739977445Z Gathering data for ns/openshift-apiserver...
2026-07-29T07:51:56.137322268Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-29T07:51:56.424160061Z Gathering data for ns/openshift-controller-manager...
2026-07-29T07:51:56.747861537Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-29T07:51:56.893601967Z Collecting kubelet logs for node ip-10-0-131-39.ec2.internal
2026-07-29T07:51:56.895417143Z Collecting kubelet logs for node ip-10-0-131-190.ec2.internal
2026-07-29T07:51:56.897249177Z Collecting kubelet logs for node ip-10-0-137-121.ec2.internal
2026-07-29T07:51:57.013866819Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-29T07:51:57.229390601Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-5kjlb namespace
2026-07-29T07:51:57.234493298Z INFO: Node performance data collection complete.
2026-07-29T07:51:57.448157541Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-29T07:51:57.732679533Z Gathering data for ns/openshift-service-ca-operator...
2026-07-29T07:51:58.034998480Z Gathering data for ns/openshift-service-ca...
2026-07-29T07:51:58.376410967Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-29T07:51:59.222315617Z Wrote inspect data to must-gather.
2026-07-29T07:51:59.222363249Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T07:51:59.222363249Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-29T07:51:59.348575360Z error: the server doesn't have a resource type "clusters"
2026-07-29T07:51:59.479452215Z Caches written to disk