2026-07-02T14:41:07.328266654Z [disk usage checker] Started
2026-07-02T14:41:07.331892717Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-02T14:41:07.541619729Z Gathering data for ns/openshift-cluster-version...
2026-07-02T14:41:07.689115157Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:41:07.779381080Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:41:07.870471255Z Gathering data for ns/default...
2026-07-02T14:41:08.209544785Z Gathering data for ns/openshift...
2026-07-02T14:41:08.637856020Z Gathering data for ns/kube-system...
2026-07-02T14:41:09.409077658Z Gathering data for ns/openshift-etcd...
2026-07-02T14:41:09.759197094Z Wrote inspect data to must-gather.
2026-07-02T14:41:09.759292441Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:41:09.759292441Z     namespaces "assisted-installer" not found
2026-07-02T14:41:10.248031284Z Waiting on subprocesses to finish execution.
2026-07-02T14:41:10.266644298Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T14:41:10.269150498Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T14:41:10.272106975Z Executing Istio gather script
2026-07-02T14:41:10.274674549Z INFO: Gathering HAProxy config files
2026-07-02T14:41:10.275587067Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T14:41:10.276049080Z INFO: Collecting host service logs for crio
2026-07-02T14:41:10.276384209Z INFO: Collecting host service logs for kubelet
2026-07-02T14:41:10.276691120Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T14:41:10.277047505Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T14:41:10.277326500Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T14:41:10.277620084Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T14:41:10.277931396Z INFO: Collecting host service logs for NetworkManager
2026-07-02T14:41:10.278227119Z INFO: Collecting host service logs for openvswitch
2026-07-02T14:41:10.278561729Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T14:41:10.278876388Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T14:41:10.279193979Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T14:41:10.279543908Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T14:41:10.410888153Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T14:41:10.683608522Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T14:41:10.683748407Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T14:41:10.935456885Z Wrote inspect data to must-gather.
2026-07-02T14:41:11.223265278Z No resources found
2026-07-02T14:41:11.224529494Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T14:41:11.241017676Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T14:41:11.268109684Z No resources found in openshift-etcd namespace.
2026-07-02T14:41:11.493038058Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T14:41:11.509562112Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T14:41:11.676116221Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T14:41:11.692729671Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-9s9bq
2026-07-02T14:41:11.692785452Z insights-runtime-extractor-425z7
2026-07-02T14:41:11.692805939Z insights-runtime-extractor-55bgg
2026-07-02T14:41:11.692825616Z insights-runtime-extractor-srnvs
2026-07-02T14:41:11.726974174Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T14:41:11.817177905Z INFO: Worker host service log collection to complete.
2026-07-02T14:41:11.823554227Z Wrote inspect data to must-gather.
2026-07-02T14:41:11.986315038Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T14:41:12.017150046Z ERROR: No running kube-apiserver pods found
2026-07-02T14:41:12.267722712Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T14:41:12.267915953Z No resources found
2026-07-02T14:41:12.370203214Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T14:41:12.375138356Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-02T14:41:12.572125999Z Wrote inspect data to must-gather.
2026-07-02T14:41:12.585491204Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:41:12.670386308Z Gathering data for ns/openshift-monitoring...
2026-07-02T14:41:12.708050334Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T14:41:12.708050334Z See 'oc rsync -h' for help and examples
2026-07-02T14:41:12.830133056Z INFO: 'previous-logs' folder not found on ip-10-0-130-230.ec2.internal, skipping...
2026-07-02T14:41:12.831123545Z tar: Removing leading `/' from member names
2026-07-02T14:41:12.836929303Z INFO: HAProxy config collection complete.
2026-07-02T14:41:12.871903204Z Wrote inspect data to must-gather.
2026-07-02T14:41:12.896420911Z Wrote inspect data to must-gather.
2026-07-02T14:41:12.940518057Z Wrote inspect data to must-gather.
2026-07-02T14:41:12.966821009Z Wrote inspect data to must-gather.
2026-07-02T14:41:13.126592844Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T14:41:13.202568074Z INFO: 'previous-logs' folder not found on ip-10-0-136-174.ec2.internal, skipping...
2026-07-02T14:41:13.243278157Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T14:41:13.267775524Z Wrote inspect data to must-gather.
2026-07-02T14:41:13.377329783Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T14:41:13.508602991Z INFO: 'previous-logs' folder not found on ip-10-0-138-160.ec2.internal, skipping...
2026-07-02T14:41:13.508637676Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T14:41:13.508671055Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T14:41:13.592575573Z Wrote inspect data to must-gather.
2026-07-02T14:41:13.640577146Z Wrote inspect data to must-gather.
2026-07-02T14:41:13.731508311Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T14:41:13.803066030Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T14:41:13.835748000Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T14:41:13.885603084Z Wrote inspect data to must-gather.
2026-07-02T14:41:14.074105160Z Wrote inspect data to must-gather.
2026-07-02T14:41:14.197128953Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T14:41:14.200345307Z INFO: INTERCONNECT MODE
2026-07-02T14:41:14.200446132Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T14:41:14.353562299Z Wrote inspect data to must-gather.
2026-07-02T14:41:14.369813327Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:41:14.393092804Z INFO: Gathering OVN_Northbound from ovnkube-node-6z75x...
2026-07-02T14:41:14.393385617Z INFO: Gathering OVN_Northbound from ovnkube-node-cnwb5...
2026-07-02T14:41:14.393638447Z INFO: Gathering OVN_Northbound from ovnkube-node-mct5z...
2026-07-02T14:41:14.393943316Z INFO: Gathering OVN_Southbound from ovnkube-node-6z75x...
2026-07-02T14:41:14.394250953Z INFO: Gathering OVN_Southbound from ovnkube-node-cnwb5...
2026-07-02T14:41:14.394724351Z INFO: Gathering OVN_Southbound from ovnkube-node-mct5z...
2026-07-02T14:41:14.517498192Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T14:41:14.729347342Z tar: Removing leading `/' from member names
2026-07-02T14:41:14.731159865Z Wrote inspect data to must-gather.
2026-07-02T14:41:14.739088109Z tar: Removing leading `/' from member names
2026-07-02T14:41:14.778454624Z tar: Removing leading `/' from member names
2026-07-02T14:41:14.782594293Z tar: Removing leading `/' from member names
2026-07-02T14:41:14.803105629Z tar: Removing leading `/' from member names
2026-07-02T14:41:14.811699867Z tar: Removing leading `/' from member names
2026-07-02T14:41:14.877615145Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T14:41:14.899767821Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T14:41:14.989344347Z Wrote inspect data to must-gather.
2026-07-02T14:41:15.111576861Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:15.179449359Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T14:41:15.193424959Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:41:15.363711870Z Wrote inspect data to must-gather.
2026-07-02T14:41:15.364382611Z INFO: Waiting for network log collection to complete ...
2026-07-02T14:41:15.364421433Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T14:41:15.364795067Z INFO: Copying ovnk databases complete.
2026-07-02T14:41:15.366686835Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-02T14:41:15.369056493Z ovnk_database_store/
2026-07-02T14:41:15.369056493Z ovnk_database_store/ovnkube-node-mct5z_sbdb
2026-07-02T14:41:15.394030103Z ovnk_database_store/ovnkube-node-cnwb5_sbdb
2026-07-02T14:41:15.399811829Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:15.407086111Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T14:41:15.417895378Z ovnk_database_store/ovnkube-node-mct5z_nbdb
2026-07-02T14:41:15.428260699Z ovnk_database_store/ovnkube-node-6z75x_nbdb
2026-07-02T14:41:15.437003957Z Gathering data for ns/openshift-network-console...
2026-07-02T14:41:15.437783017Z ovnk_database_store/ovnkube-node-6z75x_sbdb
2026-07-02T14:41:15.460751005Z ovnk_database_store/ovnkube-node-cnwb5_nbdb
2026-07-02T14:41:15.472062511Z INFO: Network log collection complete.
2026-07-02T14:41:15.649667088Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:15.654206679Z INFO: Getting status from alertmanager-main-0
2026-07-02T14:41:15.676344780Z Wrote inspect data to must-gather.
2026-07-02T14:41:15.908388303Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:15.911011656Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T14:41:15.976098018Z Gathering data for ns/openshift-console-operator...
2026-07-02T14:41:16.002575655Z Wrote inspect data to must-gather.
2026-07-02T14:41:16.128044043Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:16.306898956Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T14:41:16.310000481Z INFO: Fallback to identify the container image from release info
2026-07-02T14:41:16.365360491Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:16.369741454Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T14:41:16.411744927Z Gathering data for ns/openshift-console...
2026-07-02T14:41:16.607652280Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:16.804165306Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:16.807193604Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-02T14:41:16.991316544Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T14:41:17.005433331Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:17.037781590Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f
2026-07-02T14:41:17.189804501Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T14:41:17.241511856Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:17.245896393Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-02T14:41:17.360299884Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T14:41:17.435733259Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-02T14:41:17.436880332Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:17.608676294Z Gathering data for ns/openshift-dns-operator...
2026-07-02T14:41:17.677225202Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:17.681015292Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-02T14:41:17.869930304Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:17.964456185Z Gathering data for ns/openshift-dns...
2026-07-02T14:41:18.075511899Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:18.078213521Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-02T14:41:18.261131427Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:18.463852701Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:18.466329550Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-02T14:41:18.522984154Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T14:41:18.635353611Z Gathering data for ns/openshift-image-registry...
2026-07-02T14:41:18.665774062Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:18.874410479Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:18.876445917Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T14:41:19.053803487Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:19.184853713Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T14:41:19.281097878Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:41:19.561744176Z Gathering data for ns/openshift-ingress...
2026-07-02T14:41:19.686113610Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T14:41:19.978015858Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T14:41:20.032421302Z Done executing Istio gather script
2026-07-02T14:41:20.401843206Z Gathering data for ns/openshift-insights...
2026-07-02T14:41:20.823280995Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T14:41:21.951993338Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T14:41:22.440774429Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-02T14:41:22.719007705Z Gathering data for ns/kserve...
2026-07-02T14:41:23.072378013Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T14:41:24.194348619Z Waiting for performance profile collector pods to become ready: 7
2026-07-02T14:41:24.470293975Z Gathering data for ns/openshift-config...
2026-07-02T14:41:24.759654833Z Gathering data for ns/openshift-config-managed...
2026-07-02T14:41:25.233580756Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T14:41:25.335751053Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T14:41:25.535048562Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T14:41:25.661717570Z Collecting performance related data for node ip-10-0-130-230.ec2.internal
2026-07-02T14:41:25.669182915Z Collecting performance related data for node ip-10-0-136-174.ec2.internal
2026-07-02T14:41:25.676856505Z Collecting performance related data for node ip-10-0-138-160.ec2.internal
2026-07-02T14:41:25.891661156Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T14:41:26.261835257Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T14:41:26.691707725Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T14:41:27.067120536Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T14:41:27.431643872Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T14:41:27.446368781Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-02T14:41:27.774669814Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T14:41:28.102295639Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T14:41:28.726876466Z Gathering data for ns/openshift-multus...
2026-07-02T14:41:30.317532541Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T14:41:31.641395725Z Gathering data for ns/openshift-host-network...
2026-07-02T14:41:31.964184929Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T14:41:32.452885621Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-02T14:41:32.536233159Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T14:41:32.995811563Z Gathering data for ns/openshift-network-operator...
2026-07-02T14:41:33.387907236Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T14:41:33.693425398Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T14:41:34.189284532Z Collecting kubelet logs for node ip-10-0-130-230.ec2.internal
2026-07-02T14:41:34.191576195Z Collecting kubelet logs for node ip-10-0-136-174.ec2.internal
2026-07-02T14:41:34.193541115Z Collecting kubelet logs for node ip-10-0-138-160.ec2.internal
2026-07-02T14:41:34.196082052Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T14:41:34.577669064Z Gathering data for ns/openshift-apiserver...
2026-07-02T14:41:34.617089299Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-sd8sv namespace
2026-07-02T14:41:34.625369912Z INFO: Node performance data collection complete.
2026-07-02T14:41:35.046830359Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T14:41:35.325029673Z Gathering data for ns/openshift-controller-manager...
2026-07-02T14:41:35.614934921Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T14:41:36.276979930Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T14:41:36.593551616Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T14:41:36.921603077Z Gathering data for ns/openshift-service-ca...
2026-07-02T14:41:37.299809380Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T14:41:37.458127853Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-02T14:41:38.313682574Z Wrote inspect data to must-gather.
2026-07-02T14:41:38.313728535Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:41:38.313728535Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T14:41:38.453886493Z error: the server doesn't have a resource type "clusters"
2026-07-02T14:41:38.594467168Z Caches written to disk