2026-07-29T18:53:41.340053986Z [disk usage checker] Started
2026-07-29T18:53:41.343463046Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T18:53:41.540058488Z Gathering data for ns/openshift-cluster-version...
2026-07-29T18:53:41.691779835Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T18:53:41.782426441Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T18:53:41.868947732Z Gathering data for ns/default...
2026-07-29T18:53:42.177091643Z Gathering data for ns/openshift...
2026-07-29T18:53:42.617033907Z Gathering data for ns/kube-system...
2026-07-29T18:53:43.318684754Z Gathering data for ns/openshift-etcd...
2026-07-29T18:53:43.612795662Z Wrote inspect data to must-gather.
2026-07-29T18:53:43.612827612Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T18:53:43.612827612Z     namespaces "assisted-installer" not found
2026-07-29T18:53:44.013898681Z Waiting on subprocesses to finish execution.
2026-07-29T18:53:44.017812267Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-29T18:53:44.020386323Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-29T18:53:44.025337695Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-29T18:53:44.030956892Z INFO: Gathering HAProxy config files
2026-07-29T18:53:44.036640908Z Executing Istio gather script
2026-07-29T18:53:44.044228303Z INFO: Collecting host service logs for crio
2026-07-29T18:53:44.044611517Z INFO: Collecting host service logs for kubelet
2026-07-29T18:53:44.044991561Z INFO: Collecting host service logs for rpm-ostreed
2026-07-29T18:53:44.045372257Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-29T18:53:44.045775928Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-29T18:53:44.046249594Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-29T18:53:44.046625251Z INFO: Collecting host service logs for NetworkManager
2026-07-29T18:53:44.046988757Z INFO: Collecting host service logs for openvswitch
2026-07-29T18:53:44.047340852Z INFO: Collecting host service logs for ovs-configuration
2026-07-29T18:53:44.047707574Z INFO: Collecting host service logs for ovsdb-server
2026-07-29T18:53:44.048059468Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-29T18:53:44.048574811Z INFO: Waiting for worker host service log collection to complete ...
2026-07-29T18:53:44.070234429Z INFO: Waiting for node performance related collection to complete ...
2026-07-29T18:53:44.569487722Z No resources found in openshift-etcd namespace.
2026-07-29T18:53:44.603991319Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-29T18:53:44.622242474Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-29T18:53:44.878360689Z error: the server doesn't have a resource type "performanceprofile"
2026-07-29T18:53:44.901500929Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-29T18:53:45.176963185Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-29T18:53:45.184912734Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-29T18:53:45.184985423Z INFO: on-disk MachineConfig config collection complete.
2026-07-29T18:53:45.569770040Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-9fr66
2026-07-29T18:53:45.569770040Z insights-runtime-extractor-2kk8l
2026-07-29T18:53:45.569770040Z insights-runtime-extractor-cgxlj
2026-07-29T18:53:45.569770040Z insights-runtime-extractor-w265z
2026-07-29T18:53:45.656667564Z Wrote inspect data to must-gather.
2026-07-29T18:53:45.666772893Z INFO: "metallb-operator" not detected. Skipping.
2026-07-29T18:53:45.793168295Z No resources found
2026-07-29T18:53:45.883441112Z ERROR: No running kube-apiserver pods found
2026-07-29T18:53:45.902575406Z INFO: Worker host service log collection to complete.
2026-07-29T18:53:46.083036948Z Wrote inspect data to must-gather.
2026-07-29T18:53:46.200948815Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-29T18:53:46.225979671Z No resources found
2026-07-29T18:53:46.228453373Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-29T18:53:46.228453373Z See 'oc rsync -h' for help and examples
2026-07-29T18:53:46.262262193Z Wrote inspect data to must-gather.
2026-07-29T18:53:46.322267183Z Wrote inspect data to must-gather.
2026-07-29T18:53:46.352551230Z Wrote inspect data to must-gather.
2026-07-29T18:53:46.365923205Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T18:53:46.388535008Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T18:53:46.408902521Z INFO: 'previous-logs' folder not found on ip-10-0-132-135.ec2.internal, skipping...
2026-07-29T18:53:46.442871345Z Wrote inspect data to must-gather.
2026-07-29T18:53:46.447166058Z Gathering data for ns/openshift-monitoring...
2026-07-29T18:53:46.475992439Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-29T18:53:46.491527432Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-29T18:53:46.592702670Z Wrote inspect data to must-gather.
2026-07-29T18:53:46.716047905Z error: the server doesn't have a resource type "machineconfigs"
2026-07-29T18:53:46.764627443Z Wrote inspect data to must-gather.
2026-07-29T18:53:46.808052058Z INFO: 'previous-logs' folder not found on ip-10-0-139-60.ec2.internal, skipping...
2026-07-29T18:53:46.821670768Z tar: Removing leading `/' from member names
2026-07-29T18:53:46.825895007Z INFO: HAProxy config collection complete.
2026-07-29T18:53:46.914803343Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-29T18:53:46.989299675Z INFO: Getting rules from prometheus-k8s-0
2026-07-29T18:53:47.086911577Z Wrote inspect data to must-gather.
2026-07-29T18:53:47.114050493Z INFO: 'previous-logs' folder not found on ip-10-0-141-242.ec2.internal, skipping...
2026-07-29T18:53:47.114324082Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-29T18:53:47.114392714Z INFO: Machine Config Daemon termination log collection complete.
2026-07-29T18:53:47.205249480Z Wrote inspect data to must-gather.
2026-07-29T18:53:47.345051521Z Wrote inspect data to must-gather.
2026-07-29T18:53:47.383567052Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-29T18:53:47.429264754Z INFO: Getting status/config from prometheus-k8s-0
2026-07-29T18:53:47.602140407Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-29T18:53:47.652000195Z Wrote inspect data to must-gather.
2026-07-29T18:53:47.688517924Z INFO: INTERCONNECT MODE
2026-07-29T18:53:47.688546015Z INFO: Gathering ovn-kubernetes DBs
2026-07-29T18:53:47.861008230Z INFO: Gathering OVN_Northbound from ovnkube-node-krwk9...
2026-07-29T18:53:47.861311965Z INFO: Gathering OVN_Northbound from ovnkube-node-nldtg...
2026-07-29T18:53:47.861590815Z INFO: Gathering OVN_Northbound from ovnkube-node-rnhqn...
2026-07-29T18:53:47.861897478Z INFO: Gathering OVN_Southbound from ovnkube-node-krwk9...
2026-07-29T18:53:47.862097338Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-29T18:53:47.862116217Z INFO: Gathering OVN_Southbound from ovnkube-node-nldtg...
2026-07-29T18:53:47.862379278Z INFO: Gathering OVN_Southbound from ovnkube-node-rnhqn...
2026-07-29T18:53:47.874066389Z Wrote inspect data to must-gather.
2026-07-29T18:53:48.167358957Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T18:53:48.192637567Z tar: Removing leading `/' from member names
2026-07-29T18:53:48.212527981Z tar: Removing leading `/' from member names
2026-07-29T18:53:48.229070965Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-29T18:53:48.234633686Z tar: Removing leading `/' from member names
2026-07-29T18:53:48.240666087Z tar: Removing leading `/' from member names
2026-07-29T18:53:48.278630583Z tar: Removing leading `/' from member names
2026-07-29T18:53:48.312382651Z tar: Removing leading `/' from member names
2026-07-29T18:53:48.324226864Z Wrote inspect data to must-gather.
2026-07-29T18:53:48.515662871Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-29T18:53:48.518897757Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-29T18:53:48.577817420Z Wrote inspect data to must-gather.
2026-07-29T18:53:48.758826681Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:48.786377195Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-29T18:53:48.822118146Z W0729 18:53:48.822068    1439 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-29T18:53:48.827208391Z Wrote inspect data to must-gather.
2026-07-29T18:53:48.835515307Z INFO: Waiting for network log collection to complete ...
2026-07-29T18:53:48.835583937Z INFO: Waiting for ovnk database copies to complete ...
2026-07-29T18:53:48.836111333Z INFO: Copying ovnk databases complete.
2026-07-29T18:53:48.841093709Z 2.8M	must-gather/network_logs/ovnk_database_store
2026-07-29T18:53:48.843568511Z ovnk_database_store/
2026-07-29T18:53:48.843568511Z ovnk_database_store/ovnkube-node-krwk9_nbdb
2026-07-29T18:53:48.851362151Z ovnk_database_store/ovnkube-node-rnhqn_sbdb
2026-07-29T18:53:48.862895249Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T18:53:48.868738349Z ovnk_database_store/ovnkube-node-krwk9_sbdb
2026-07-29T18:53:48.887617934Z ovnk_database_store/ovnkube-node-nldtg_sbdb
2026-07-29T18:53:48.904127598Z ovnk_database_store/ovnkube-node-nldtg_nbdb
2026-07-29T18:53:48.913040219Z ovnk_database_store/ovnkube-node-rnhqn_nbdb
2026-07-29T18:53:48.925233981Z INFO: Network log collection complete.
2026-07-29T18:53:49.013036506Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:49.016456815Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-29T18:53:49.068033474Z Gathering data for ns/openshift-network-console...
2026-07-29T18:53:49.147846005Z Wrote inspect data to must-gather.
2026-07-29T18:53:49.221257509Z INFO: Getting status from alertmanager-main-0
2026-07-29T18:53:49.233225752Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:49.469931458Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:49.472154983Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-29T18:53:49.490750840Z Wrote inspect data to must-gather.
2026-07-29T18:53:49.582339440Z Gathering data for ns/openshift-console-operator...
2026-07-29T18:53:49.677934837Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:49.782753551Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-29T18:53:49.785061974Z INFO: Fallback to identify the container image from release info
2026-07-29T18:53:49.880513398Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:49.882883430Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-29T18:53:50.002625957Z Gathering data for ns/openshift-console...
2026-07-29T18:53:50.126103580Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:50.308583514Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:50.311756123Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-29T18:53:50.490952123Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:50.517200035Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-29T18:53:50.596912495Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-29T18:53:50.707724586Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:50.710565982Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-29T18:53:50.738301254Z daemonset.apps/perf-node-gather-daemonset created
2026-07-29T18:53:50.898221616Z Waiting for performance profile collector pods to become ready: 1
2026-07-29T18:53:50.909692258Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:51.095441868Z Gathering data for ns/openshift-dns-operator...
2026-07-29T18:53:51.106534788Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:51.109841025Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-29T18:53:51.315585828Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:51.421576465Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T18:53:51.429113076Z Gathering data for ns/openshift-dns...
2026-07-29T18:53:51.501901707Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:51.504645056Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-29T18:53:51.663755912Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:51.854304224Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:51.856723784Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-29T18:53:52.047822960Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:52.048193078Z Gathering data for ns/openshift-image-registry...
2026-07-29T18:53:52.048773374Z Waiting for performance profile collector pods to become ready: 2
2026-07-29T18:53:52.240594093Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:52.243357057Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-29T18:53:52.410175823Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:52.549910150Z Gathering data for ns/openshift-ingress-operator...
2026-07-29T18:53:52.620129027Z Wrote inspect data to /must-gather/istio.
2026-07-29T18:53:52.928316041Z Gathering data for ns/openshift-ingress...
2026-07-29T18:53:53.186078963Z Waiting for performance profile collector pods to become ready: 3
2026-07-29T18:53:53.290210439Z Gathering data for ns/openshift-ingress-canary...
2026-07-29T18:53:53.318962178Z Done executing Istio gather script
2026-07-29T18:53:53.667404881Z Gathering data for ns/openshift-insights...
2026-07-29T18:53:54.315201422Z Waiting for performance profile collector pods to become ready: 4
2026-07-29T18:53:55.428598424Z Waiting for performance profile collector pods to become ready: 5
2026-07-29T18:53:55.853944559Z Gathering data for ns/kserve...
2026-07-29T18:53:56.426075318Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T18:53:56.543802154Z Waiting for performance profile collector pods to become ready: 6
2026-07-29T18:53:57.073266864Z Gathering data for ns/openshift-config...
2026-07-29T18:53:57.322407255Z Gathering data for ns/openshift-config-managed...
2026-07-29T18:53:57.680078833Z Waiting for performance profile collector pods to become ready: 7
2026-07-29T18:53:57.797463015Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-29T18:53:58.037454388Z Gathering data for ns/openshift-kube-apiserver...
2026-07-29T18:53:58.323434154Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-29T18:53:58.635216819Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-29T18:53:58.802157959Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-29T18:53:58.931968124Z Gathering data for ns/openshift-kube-scheduler...
2026-07-29T18:53:59.071217383Z Collecting performance related data for node ip-10-0-141-242.ec2.internal
2026-07-29T18:53:59.079366811Z Collecting performance related data for node ip-10-0-132-135.ec2.internal
2026-07-29T18:53:59.088368128Z Collecting performance related data for node ip-10-0-139-60.ec2.internal
2026-07-29T18:53:59.243456162Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-29T18:53:59.585509154Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-29T18:54:00.025809614Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-29T18:54:00.401061193Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-29T18:54:00.943963609Z Gathering data for ns/openshift-multus...
2026-07-29T18:54:01.431641401Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T18:54:02.424947196Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-29T18:54:03.682323662Z Gathering data for ns/openshift-host-network...
2026-07-29T18:54:03.991893889Z Gathering data for ns/openshift-network-diagnostics...
2026-07-29T18:54:04.516271726Z Gathering data for ns/openshift-network-node-identity...
2026-07-29T18:54:04.936350883Z Gathering data for ns/openshift-network-operator...
2026-07-29T18:54:05.317146996Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-29T18:54:05.599442848Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-29T18:54:06.062609160Z Gathering data for ns/openshift-apiserver-operator...
2026-07-29T18:54:06.367401559Z Gathering data for ns/openshift-apiserver...
2026-07-29T18:54:06.436351292Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T18:54:06.741916079Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-29T18:54:07.026055859Z Gathering data for ns/openshift-controller-manager...
2026-07-29T18:54:07.223907303Z Collecting kubelet logs for node ip-10-0-141-242.ec2.internal
2026-07-29T18:54:07.225990248Z Collecting kubelet logs for node ip-10-0-132-135.ec2.internal
2026-07-29T18:54:07.228330372Z Collecting kubelet logs for node ip-10-0-139-60.ec2.internal
2026-07-29T18:54:07.331628669Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-29T18:54:07.572330241Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-gd9l8 namespace
2026-07-29T18:54:07.578350181Z INFO: Node performance data collection complete.
2026-07-29T18:54:08.016839109Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-29T18:54:08.329004810Z Gathering data for ns/openshift-service-ca-operator...
2026-07-29T18:54:08.623394208Z Gathering data for ns/openshift-service-ca...
2026-07-29T18:54:08.956123001Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-29T18:54:09.813560480Z Wrote inspect data to must-gather.
2026-07-29T18:54:09.813593083Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T18:54:09.813593083Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-29T18:54:09.948725954Z error: the server doesn't have a resource type "clusters"
2026-07-29T18:54:10.087650548Z Caches written to disk