2026-07-29T17:41:22.498289016Z [disk usage checker] Started
2026-07-29T17:41:22.501666542Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-29T17:41:22.685354544Z Gathering data for ns/openshift-cluster-version...
2026-07-29T17:41:22.787401493Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T17:41:22.867568629Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T17:41:22.944408914Z Gathering data for ns/default...
2026-07-29T17:41:23.251151331Z Gathering data for ns/openshift...
2026-07-29T17:41:23.633594994Z Gathering data for ns/kube-system...
2026-07-29T17:41:24.338625612Z Gathering data for ns/openshift-etcd...
2026-07-29T17:41:24.663994679Z Wrote inspect data to must-gather.
2026-07-29T17:41:24.664025309Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T17:41:24.664025309Z     namespaces "assisted-installer" not found
2026-07-29T17:41:25.047395660Z Waiting on subprocesses to finish execution.
2026-07-29T17:41:25.056907703Z INFO: Gathering HAProxy config files
2026-07-29T17:41:25.060949882Z Executing Istio gather script
2026-07-29T17:41:25.064022169Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-29T17:41:25.069146993Z INFO: Collecting host service logs for crio
2026-07-29T17:41:25.069362370Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-29T17:41:25.069419799Z INFO: Collecting host service logs for kubelet
2026-07-29T17:41:25.069730001Z INFO: Collecting host service logs for rpm-ostreed
2026-07-29T17:41:25.070059654Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-29T17:41:25.070344952Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-29T17:41:25.070656499Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-29T17:41:25.070993468Z INFO: Collecting host service logs for NetworkManager
2026-07-29T17:41:25.071283745Z INFO: Collecting host service logs for openvswitch
2026-07-29T17:41:25.071584792Z INFO: Collecting host service logs for ovs-configuration
2026-07-29T17:41:25.071905515Z INFO: Collecting host service logs for ovsdb-server
2026-07-29T17:41:25.072235698Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-29T17:41:25.072629405Z INFO: Waiting for worker host service log collection to complete ...
2026-07-29T17:41:25.076361903Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-29T17:41:25.228224044Z INFO: Waiting for node performance related collection to complete ...
2026-07-29T17:41:25.301094853Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-29T17:41:25.794244806Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-29T17:41:25.899023668Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-dwj8t
2026-07-29T17:41:25.899091756Z insights-runtime-extractor-nl5v6
2026-07-29T17:41:25.899113180Z insights-runtime-extractor-ssr7f
2026-07-29T17:41:25.899132245Z insights-runtime-extractor-w8b2l
2026-07-29T17:41:26.003001979Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-29T17:41:26.003043173Z INFO: on-disk MachineConfig config collection complete.
2026-07-29T17:41:26.019160204Z Wrote inspect data to must-gather.
2026-07-29T17:41:26.071534773Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-29T17:41:26.313008398Z No resources found in openshift-etcd namespace.
2026-07-29T17:41:26.314414597Z INFO: Worker host service log collection to complete.
2026-07-29T17:41:26.370753320Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-29T17:41:26.370753320Z See 'oc rsync -h' for help and examples
2026-07-29T17:41:26.382897796Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-29T17:41:26.388307661Z ERROR: No running kube-apiserver pods found
2026-07-29T17:41:26.416477951Z INFO: "metallb-operator" not detected. Skipping.
2026-07-29T17:41:26.422500362Z No resources found
2026-07-29T17:41:26.441131588Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-29T17:41:26.593063188Z Gathering data for ns/openshift-monitoring...
2026-07-29T17:41:26.651946059Z error: the server doesn't have a resource type "performanceprofile"
2026-07-29T17:41:26.777004180Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-29T17:41:26.804316835Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-29T17:41:26.989970762Z Wrote inspect data to must-gather.
2026-07-29T17:41:27.115704044Z Wrote inspect data to must-gather.
2026-07-29T17:41:27.221871403Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T17:41:27.350884685Z Wrote inspect data to must-gather.
2026-07-29T17:41:27.374343823Z No resources found
2026-07-29T17:41:27.401994266Z tar: Removing leading `/' from member names
2026-07-29T17:41:27.409028349Z INFO: HAProxy config collection complete.
2026-07-29T17:41:27.410066229Z Wrote inspect data to must-gather.
2026-07-29T17:41:27.417495064Z Wrote inspect data to must-gather.
2026-07-29T17:41:27.478656081Z INFO: 'previous-logs' folder not found on ip-10-0-129-165.ec2.internal, skipping...
2026-07-29T17:41:27.507870938Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-29T17:41:27.513241677Z Wrote inspect data to must-gather.
2026-07-29T17:41:27.688019846Z Wrote inspect data to must-gather.
2026-07-29T17:41:27.706973001Z error: the server doesn't have a resource type "machineconfigs"
2026-07-29T17:41:27.744609313Z INFO: Getting rules from prometheus-k8s-0
2026-07-29T17:41:27.794143872Z INFO: 'previous-logs' folder not found on ip-10-0-137-138.ec2.internal, skipping...
2026-07-29T17:41:27.797250183Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-29T17:41:27.895169552Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-29T17:41:27.969708372Z Wrote inspect data to must-gather.
2026-07-29T17:41:28.057348352Z INFO: 'previous-logs' folder not found on ip-10-0-139-166.ec2.internal, skipping...
2026-07-29T17:41:28.057388159Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-29T17:41:28.057417677Z INFO: Machine Config Daemon termination log collection complete.
2026-07-29T17:41:28.139581800Z Wrote inspect data to must-gather.
2026-07-29T17:41:28.160245282Z INFO: Getting status/config from prometheus-k8s-0
2026-07-29T17:41:28.227762856Z Wrote inspect data to must-gather.
2026-07-29T17:41:28.304542594Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-29T17:41:28.484874029Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T17:41:28.500691294Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-29T17:41:28.525433822Z INFO: INTERCONNECT MODE
2026-07-29T17:41:28.525529582Z INFO: Gathering ovn-kubernetes DBs
2026-07-29T17:41:28.543172875Z Wrote inspect data to must-gather.
2026-07-29T17:41:28.567106191Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-29T17:41:28.701572710Z INFO: Gathering OVN_Northbound from ovnkube-node-9m7h5...
2026-07-29T17:41:28.701923144Z INFO: Gathering OVN_Northbound from ovnkube-node-h5m4m...
2026-07-29T17:41:28.702193126Z INFO: Gathering OVN_Northbound from ovnkube-node-wfqvb...
2026-07-29T17:41:28.702481747Z INFO: Gathering OVN_Southbound from ovnkube-node-9m7h5...
2026-07-29T17:41:28.702734484Z INFO: Gathering OVN_Southbound from ovnkube-node-h5m4m...
2026-07-29T17:41:28.703026849Z INFO: Gathering OVN_Southbound from ovnkube-node-wfqvb...
2026-07-29T17:41:28.712137095Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:28.825077286Z Wrote inspect data to must-gather.
2026-07-29T17:41:28.907876943Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-29T17:41:29.022039738Z tar: Removing leading `/' from member names
2026-07-29T17:41:29.039646856Z tar: Removing leading `/' from member names
2026-07-29T17:41:29.055001264Z tar: Removing leading `/' from member names
2026-07-29T17:41:29.056425760Z tar: Removing leading `/' from member names
2026-07-29T17:41:29.057901233Z tar: Removing leading `/' from member names
2026-07-29T17:41:29.058719698Z tar: Removing leading `/' from member names
2026-07-29T17:41:29.150573549Z W0729 17:41:29.150533    1371 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-29T17:41:29.157523451Z Wrote inspect data to must-gather.
2026-07-29T17:41:29.159695213Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:29.162684529Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-29T17:41:29.217946716Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-29T17:41:29.251111859Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T17:41:29.406629445Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:29.484247102Z Gathering data for ns/openshift-network-console...
2026-07-29T17:41:29.507468406Z Wrote inspect data to must-gather.
2026-07-29T17:41:29.510674643Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-29T17:41:29.636076650Z INFO: Waiting for network log collection to complete ...
2026-07-29T17:41:29.636194489Z INFO: Waiting for ovnk database copies to complete ...
2026-07-29T17:41:29.636740522Z INFO: Copying ovnk databases complete.
2026-07-29T17:41:29.637144245Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:29.638505867Z 2.4M	must-gather/network_logs/ovnk_database_store
2026-07-29T17:41:29.640090856Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-29T17:41:29.640759538Z ovnk_database_store/
2026-07-29T17:41:29.640759538Z ovnk_database_store/ovnkube-node-h5m4m_sbdb
2026-07-29T17:41:29.659022554Z ovnk_database_store/ovnkube-node-wfqvb_sbdb
2026-07-29T17:41:29.675237899Z ovnk_database_store/ovnkube-node-wfqvb_nbdb
2026-07-29T17:41:29.679918526Z ovnk_database_store/ovnkube-node-h5m4m_nbdb
2026-07-29T17:41:29.685643065Z ovnk_database_store/ovnkube-node-9m7h5_nbdb
2026-07-29T17:41:29.690899344Z ovnk_database_store/ovnkube-node-9m7h5_sbdb
2026-07-29T17:41:29.705856264Z INFO: Network log collection complete.
2026-07-29T17:41:29.828376814Z Wrote inspect data to must-gather.
2026-07-29T17:41:29.835041376Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:29.923264663Z INFO: Getting status from alertmanager-main-0
2026-07-29T17:41:29.981766619Z Gathering data for ns/openshift-console-operator...
2026-07-29T17:41:30.072062479Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:30.074902156Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-29T17:41:30.141996559Z Wrote inspect data to must-gather.
2026-07-29T17:41:30.306850148Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:30.410677056Z Gathering data for ns/openshift-console...
2026-07-29T17:41:30.430616743Z Wrote inspect data to must-gather.
2026-07-29T17:41:30.513408356Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:30.516875326Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-29T17:41:30.687615547Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:30.693057113Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-29T17:41:30.695324377Z INFO: Fallback to identify the container image from release info
2026-07-29T17:41:30.895687292Z Wrote inspect data to /must-gather/istio.
2026-07-29T17:41:30.933937683Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-29T17:41:31.442436670Z Gathering data for ns/openshift-dns-operator...
2026-07-29T17:41:31.535670973Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-29T17:41:31.578357108Z Done executing Istio gather script
2026-07-29T17:41:31.654550449Z daemonset.apps/perf-node-gather-daemonset created
2026-07-29T17:41:31.763517367Z Gathering data for ns/openshift-dns...
2026-07-29T17:41:31.781767768Z Waiting for performance profile collector pods to become ready: 1
2026-07-29T17:41:32.343143189Z Gathering data for ns/openshift-image-registry...
2026-07-29T17:41:32.512902511Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-29T17:41:32.770384630Z Gathering data for ns/openshift-ingress-operator...
2026-07-29T17:41:32.897275549Z Waiting for performance profile collector pods to become ready: 2
2026-07-29T17:41:33.070765716Z Gathering data for ns/openshift-ingress...
2026-07-29T17:41:33.368488166Z Gathering data for ns/openshift-ingress-canary...
2026-07-29T17:41:33.726759824Z Gathering data for ns/openshift-insights...
2026-07-29T17:41:34.012592474Z Waiting for performance profile collector pods to become ready: 3
2026-07-29T17:41:35.130675282Z Waiting for performance profile collector pods to become ready: 4
2026-07-29T17:41:35.514998091Z Gathering data for ns/openshift-config...
2026-07-29T17:41:35.762081543Z Gathering data for ns/openshift-config-managed...
2026-07-29T17:41:36.160107560Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-29T17:41:36.256987691Z Waiting for performance profile collector pods to become ready: 5
2026-07-29T17:41:36.440367364Z Gathering data for ns/openshift-kube-apiserver...
2026-07-29T17:41:36.673835573Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-29T17:41:36.932405299Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-29T17:41:37.192202603Z Gathering data for ns/openshift-kube-scheduler...
2026-07-29T17:41:37.378411546Z Waiting for performance profile collector pods to become ready: 6
2026-07-29T17:41:37.432550835Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-29T17:41:37.517481586Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-29T17:41:37.685912799Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-29T17:41:37.968165255Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-29T17:41:38.240203214Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-29T17:41:38.503521275Z Waiting for performance profile collector pods to become ready: 7
2026-07-29T17:41:38.741515313Z Gathering data for ns/openshift-multus...
2026-07-29T17:41:39.626991778Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-29T17:41:39.868834392Z Collecting performance related data for node ip-10-0-137-138.ec2.internal
2026-07-29T17:41:39.877740888Z Collecting performance related data for node ip-10-0-139-166.ec2.internal
2026-07-29T17:41:39.886260881Z Collecting performance related data for node ip-10-0-129-165.ec2.internal
2026-07-29T17:41:40.240730014Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-29T17:41:41.554767949Z Gathering data for ns/openshift-host-network...
2026-07-29T17:41:41.909237096Z Gathering data for ns/openshift-network-diagnostics...
2026-07-29T17:41:42.453355334Z Gathering data for ns/openshift-network-node-identity...
2026-07-29T17:41:42.525330922Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-29T17:41:42.881135635Z Gathering data for ns/openshift-network-operator...
2026-07-29T17:41:43.229420680Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-29T17:41:43.469531781Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-29T17:41:43.980851409Z Gathering data for ns/openshift-apiserver-operator...
2026-07-29T17:41:44.312659669Z Gathering data for ns/openshift-apiserver...
2026-07-29T17:41:44.703255957Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-29T17:41:44.971714288Z Gathering data for ns/openshift-controller-manager...
2026-07-29T17:41:45.295565384Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-29T17:41:45.946065005Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-29T17:41:46.213683864Z Gathering data for ns/openshift-service-ca-operator...
2026-07-29T17:41:46.504582058Z Gathering data for ns/openshift-service-ca...
2026-07-29T17:41:46.848812614Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-29T17:41:47.531747258Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-29T17:41:47.743145889Z Collecting kubelet logs for node ip-10-0-137-138.ec2.internal
2026-07-29T17:41:47.744950864Z Collecting kubelet logs for node ip-10-0-139-166.ec2.internal
2026-07-29T17:41:47.747176268Z Collecting kubelet logs for node ip-10-0-129-165.ec2.internal
2026-07-29T17:41:47.763551676Z Wrote inspect data to must-gather.
2026-07-29T17:41:47.764226604Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T17:41:47.764226604Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-29T17:41:48.035243151Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-vkrrk namespace
2026-07-29T17:41:48.040612418Z INFO: Node performance data collection complete.
2026-07-29T17:41:48.163536515Z error: the server doesn't have a resource type "clusters"
2026-07-29T17:41:48.308871955Z Caches written to disk