2026-07-31T15:39:24.275811890Z [disk usage checker] Started
2026-07-31T15:39:24.279120186Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-31T15:39:24.470747860Z Gathering data for ns/openshift-cluster-version...
2026-07-31T15:39:24.610773324Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-31T15:39:24.704236507Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-31T15:39:24.791083451Z Gathering data for ns/default...
2026-07-31T15:39:25.130405515Z Gathering data for ns/openshift...
2026-07-31T15:39:25.560466892Z Gathering data for ns/kube-system...
2026-07-31T15:39:26.405598993Z Gathering data for ns/openshift-etcd...
2026-07-31T15:39:26.779398726Z Wrote inspect data to must-gather.
2026-07-31T15:39:26.779426020Z error: inspection completed with the errors occurred while gathering data:
2026-07-31T15:39:26.779426020Z     namespaces "assisted-installer" not found
2026-07-31T15:39:27.083468046Z Waiting on subprocesses to finish execution.
2026-07-31T15:39:27.090547090Z Executing Istio gather script
2026-07-31T15:39:27.095587249Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-31T15:39:27.096608963Z INFO: Gathering HAProxy config files
2026-07-31T15:39:27.102781515Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-31T15:39:27.120221119Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-31T15:39:27.142361808Z INFO: Waiting for node performance related collection to complete ...
2026-07-31T15:39:27.302481963Z INFO: Collecting host service logs for crio
2026-07-31T15:39:27.302912554Z INFO: Collecting host service logs for kubelet
2026-07-31T15:39:27.303274611Z INFO: Collecting host service logs for rpm-ostreed
2026-07-31T15:39:27.303605679Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-31T15:39:27.303949730Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-31T15:39:27.304311699Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-31T15:39:27.304639550Z INFO: Collecting host service logs for NetworkManager
2026-07-31T15:39:27.304980382Z INFO: Collecting host service logs for openvswitch
2026-07-31T15:39:27.305304695Z INFO: Collecting host service logs for ovs-configuration
2026-07-31T15:39:27.305625764Z INFO: Collecting host service logs for ovsdb-server
2026-07-31T15:39:27.306014631Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-31T15:39:27.306413261Z INFO: Waiting for worker host service log collection to complete ...
2026-07-31T15:39:27.425474596Z No resources found in openshift-etcd namespace.
2026-07-31T15:39:27.782974453Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-31T15:39:27.824699021Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-31T15:39:27.837966693Z INFO: Collecting Insights Archives from insights-operator-77696685b-8zj96
2026-07-31T15:39:27.838005762Z insights-runtime-extractor-5q6ms
2026-07-31T15:39:27.838026740Z insights-runtime-extractor-bglfl
2026-07-31T15:39:27.838046671Z insights-runtime-extractor-wq62m
2026-07-31T15:39:28.038870004Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-31T15:39:28.180535929Z Wrote inspect data to must-gather.
2026-07-31T15:39:28.355404831Z No resources found
2026-07-31T15:39:28.365677995Z error: the server doesn't have a resource type "performanceprofile"
2026-07-31T15:39:28.579058154Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-31T15:39:28.580937342Z INFO: "metallb-operator" not detected. Skipping.
2026-07-31T15:39:28.660301073Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-31T15:39:28.660301073Z See 'oc rsync -h' for help and examples
2026-07-31T15:39:28.789784127Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-31T15:39:28.789893585Z INFO: on-disk MachineConfig config collection complete.
2026-07-31T15:39:29.009121449Z ERROR: No running kube-apiserver pods found
2026-07-31T15:39:29.033190975Z INFO: Worker host service log collection to complete.
2026-07-31T15:39:29.286459257Z Wrote inspect data to must-gather.
2026-07-31T15:39:29.291056061Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-31T15:39:29.356098705Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-31T15:39:29.418947599Z Wrote inspect data to must-gather.
2026-07-31T15:39:29.439386477Z INFO: 'previous-logs' folder not found on ip-10-0-135-216.ec2.internal, skipping...
2026-07-31T15:39:29.491294870Z No resources found
2026-07-31T15:39:29.502100344Z Wrote inspect data to must-gather.
2026-07-31T15:39:29.503272853Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-31T15:39:29.648222185Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-31T15:39:29.681357682Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-31T15:39:29.828813361Z Wrote inspect data to must-gather.
2026-07-31T15:39:29.919464426Z error: the server doesn't have a resource type "machineconfigs"
2026-07-31T15:39:29.925509532Z Wrote inspect data to must-gather.
2026-07-31T15:39:29.944321869Z Wrote inspect data to must-gather.
2026-07-31T15:39:30.096073281Z Gathering data for ns/openshift-monitoring...
2026-07-31T15:39:30.111160328Z INFO: 'previous-logs' folder not found on ip-10-0-136-197.ec2.internal, skipping...
2026-07-31T15:39:30.122597405Z tar: Removing leading `/' from member names
2026-07-31T15:39:30.126438016Z INFO: HAProxy config collection complete.
2026-07-31T15:39:30.187536031Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-31T15:39:30.207053092Z Wrote inspect data to must-gather.
2026-07-31T15:39:30.287047086Z INFO: Getting rules from prometheus-k8s-0
2026-07-31T15:39:30.399094099Z INFO: 'previous-logs' folder not found on ip-10-0-143-167.ec2.internal, skipping...
2026-07-31T15:39:30.399201944Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-31T15:39:30.399257357Z INFO: Machine Config Daemon termination log collection complete.
2026-07-31T15:39:30.462575087Z Wrote inspect data to must-gather.
2026-07-31T15:39:30.518281626Z Wrote inspect data to must-gather.
2026-07-31T15:39:30.662262380Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-31T15:39:30.712560450Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-31T15:39:30.770916480Z INFO: Getting status/config from prometheus-k8s-0
2026-07-31T15:39:30.782050411Z Wrote inspect data to must-gather.
2026-07-31T15:39:30.928466206Z Wrote inspect data to must-gather.
2026-07-31T15:39:31.090435232Z INFO: INTERCONNECT MODE
2026-07-31T15:39:31.090478575Z INFO: Gathering ovn-kubernetes DBs
2026-07-31T15:39:31.175442975Z Wrote inspect data to must-gather.
2026-07-31T15:39:31.230415469Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-31T15:39:31.257312167Z INFO: Gathering OVN_Northbound from ovnkube-node-76j6l...
2026-07-31T15:39:31.257660421Z INFO: Gathering OVN_Northbound from ovnkube-node-b95kt...
2026-07-31T15:39:31.257936354Z INFO: Gathering OVN_Northbound from ovnkube-node-qnsmj...
2026-07-31T15:39:31.258218895Z INFO: Gathering OVN_Southbound from ovnkube-node-76j6l...
2026-07-31T15:39:31.258496100Z INFO: Gathering OVN_Southbound from ovnkube-node-b95kt...
2026-07-31T15:39:31.258803456Z INFO: Gathering OVN_Southbound from ovnkube-node-qnsmj...
2026-07-31T15:39:31.537196608Z Wrote inspect data to must-gather.
2026-07-31T15:39:31.635037189Z tar: Removing leading `/' from member names
2026-07-31T15:39:31.669404015Z tar: Removing leading `/' from member names
2026-07-31T15:39:31.669595818Z tar: Removing leading `/' from member names
2026-07-31T15:39:31.683778440Z tar: Removing leading `/' from member names
2026-07-31T15:39:31.691984399Z tar: Removing leading `/' from member names
2026-07-31T15:39:31.694473656Z tar: Removing leading `/' from member names
2026-07-31T15:39:31.698682944Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-31T15:39:31.797436566Z Wrote inspect data to must-gather.
2026-07-31T15:39:31.978487320Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-31T15:39:32.014975126Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-31T15:39:32.019335151Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-31T15:39:32.063282192Z Wrote inspect data to must-gather.
2026-07-31T15:39:32.154767287Z INFO: Waiting for network log collection to complete ...
2026-07-31T15:39:32.154803240Z INFO: Waiting for ovnk database copies to complete ...
2026-07-31T15:39:32.155191703Z INFO: Copying ovnk databases complete.
2026-07-31T15:39:32.156741909Z 3.9M	must-gather/network_logs/ovnk_database_store
2026-07-31T15:39:32.159063588Z ovnk_database_store/
2026-07-31T15:39:32.159063588Z ovnk_database_store/ovnkube-node-qnsmj_nbdb
2026-07-31T15:39:32.171521317Z ovnk_database_store/ovnkube-node-qnsmj_sbdb
2026-07-31T15:39:32.183476641Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:32.198472406Z ovnk_database_store/ovnkube-node-b95kt_nbdb
2026-07-31T15:39:32.208895813Z ovnk_database_store/ovnkube-node-76j6l_nbdb
2026-07-31T15:39:32.217329854Z ovnk_database_store/ovnkube-node-b95kt_sbdb
2026-07-31T15:39:32.243983841Z ovnk_database_store/ovnkube-node-76j6l_sbdb
2026-07-31T15:39:32.268966205Z INFO: Network log collection complete.
2026-07-31T15:39:32.306747050Z Wrote inspect data to must-gather.
2026-07-31T15:39:32.321358291Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-31T15:39:32.400557217Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:32.402789474Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-31T15:39:32.546631296Z Wrote inspect data to must-gather.
2026-07-31T15:39:32.615394019Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:32.689588857Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-31T15:39:32.771589164Z INFO: Getting status from alertmanager-main-0
2026-07-31T15:39:32.867157758Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-31T15:39:32.869082439Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:32.872357729Z INFO: Fallback to identify the container image from release info
2026-07-31T15:39:32.872946717Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-31T15:39:32.927696803Z Gathering data for ns/openshift-network-console...
2026-07-31T15:39:33.083694620Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:33.279786263Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:33.282481496Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-31T15:39:33.401722846Z Gathering data for ns/openshift-console-operator...
2026-07-31T15:39:33.530956109Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:33.547313857Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2eb1033f55d7129b37ecfc3e4e9a306e42e5ab3dd7d2fe0ff08c8b45ea2b675
2026-07-31T15:39:33.669934285Z daemonset.apps/perf-node-gather-daemonset created
2026-07-31T15:39:33.742351295Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:33.745442100Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-31T15:39:33.820311457Z Waiting for performance profile collector pods to become ready: 1
2026-07-31T15:39:33.897666559Z Gathering data for ns/openshift-console...
2026-07-31T15:39:33.931631213Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:34.139479657Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:34.143206276Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-31T15:39:34.306527888Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-31T15:39:34.343546289Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:34.488492440Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-31T15:39:34.547934881Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:34.551434000Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-31T15:39:34.728349477Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:34.947196393Z Wrote inspect data to /must-gather/istio.
2026-07-31T15:39:34.981549191Z Waiting for performance profile collector pods to become ready: 2
2026-07-31T15:39:35.075893896Z Gathering data for ns/openshift-dns-operator...
2026-07-31T15:39:35.406135619Z Gathering data for ns/openshift-dns...
2026-07-31T15:39:35.667344944Z Done executing Istio gather script
2026-07-31T15:39:36.113706854Z Waiting for performance profile collector pods to become ready: 3
2026-07-31T15:39:36.211701351Z Gathering data for ns/openshift-image-registry...
2026-07-31T15:39:36.780192481Z Gathering data for ns/openshift-ingress-operator...
2026-07-31T15:39:37.184063510Z Gathering data for ns/openshift-ingress...
2026-07-31T15:39:37.249769868Z Waiting for performance profile collector pods to become ready: 4
2026-07-31T15:39:37.532726992Z Gathering data for ns/openshift-ingress-canary...
2026-07-31T15:39:37.956756607Z Gathering data for ns/openshift-insights...
2026-07-31T15:39:38.391717735Z Waiting for performance profile collector pods to become ready: 5
2026-07-31T15:39:39.311242656Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-31T15:39:39.528456976Z Waiting for performance profile collector pods to become ready: 6
2026-07-31T15:39:40.654637948Z Waiting for performance profile collector pods to become ready: 7
2026-07-31T15:39:40.661797691Z Gathering data for ns/kserve...
2026-07-31T15:39:41.777069702Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-31T15:39:42.033247268Z Collecting performance related data for node ip-10-0-143-167.ec2.internal
2026-07-31T15:39:42.040086228Z Collecting performance related data for node ip-10-0-135-216.ec2.internal
2026-07-31T15:39:42.047542830Z Collecting performance related data for node ip-10-0-136-197.ec2.internal
2026-07-31T15:39:42.681247567Z Gathering data for ns/openshift-config...
2026-07-31T15:39:43.088239785Z Gathering data for ns/openshift-config-managed...
2026-07-31T15:39:43.658526135Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-31T15:39:44.119217002Z Gathering data for ns/openshift-kube-apiserver...
2026-07-31T15:39:44.315709160Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-31T15:39:44.443017902Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-31T15:39:44.789863066Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-31T15:39:45.212813473Z Gathering data for ns/openshift-kube-scheduler...
2026-07-31T15:39:45.568306072Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-31T15:39:45.913570792Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-31T15:39:46.249925699Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-31T15:39:46.582410252Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-31T15:39:47.244859385Z Gathering data for ns/openshift-multus...
2026-07-31T15:39:48.902495726Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-31T15:39:49.320197117Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-31T15:39:50.222224561Z Gathering data for ns/openshift-host-network...
2026-07-31T15:39:50.569161687Z Gathering data for ns/openshift-network-diagnostics...
2026-07-31T15:39:50.614141348Z Collecting kubelet logs for node ip-10-0-143-167.ec2.internal
2026-07-31T15:39:50.615883406Z Collecting kubelet logs for node ip-10-0-135-216.ec2.internal
2026-07-31T15:39:50.617526008Z Collecting kubelet logs for node ip-10-0-136-197.ec2.internal
2026-07-31T15:39:50.986170279Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-cd56f namespace
2026-07-31T15:39:50.992578344Z INFO: Node performance data collection complete.
2026-07-31T15:39:51.187366542Z Gathering data for ns/openshift-network-node-identity...
2026-07-31T15:39:51.619695252Z Gathering data for ns/openshift-network-operator...
2026-07-31T15:39:52.002186703Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-31T15:39:52.304558768Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-31T15:39:52.802180276Z Gathering data for ns/openshift-apiserver-operator...
2026-07-31T15:39:53.127145208Z Gathering data for ns/openshift-apiserver...
2026-07-31T15:39:53.568534212Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-31T15:39:53.863683073Z Gathering data for ns/openshift-controller-manager...
2026-07-31T15:39:54.135804721Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-31T15:39:54.324801261Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-31T15:39:54.784301437Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-31T15:39:55.093587629Z Gathering data for ns/openshift-service-ca-operator...
2026-07-31T15:39:55.411948555Z Gathering data for ns/openshift-service-ca...
2026-07-31T15:39:55.796365945Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-31T15:39:56.684515358Z Wrote inspect data to must-gather.
2026-07-31T15:39:56.684560541Z error: inspection completed with the errors occurred while gathering data:
2026-07-31T15:39:56.684560541Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-31T15:39:56.810576926Z error: the server doesn't have a resource type "clusters"
2026-07-31T15:39:56.953012949Z Caches written to disk