2026-07-09T18:31:51.806500473Z [disk usage checker] Started
2026-07-09T18:31:51.810364349Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-09T18:31:52.013791391Z Gathering data for ns/openshift-cluster-version...
2026-07-09T18:31:52.134636368Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:31:52.237478691Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:31:52.323039371Z Gathering data for ns/default...
2026-07-09T18:31:52.644556257Z Gathering data for ns/openshift...
2026-07-09T18:31:53.139125239Z Gathering data for ns/kube-system...
2026-07-09T18:31:53.911025053Z Gathering data for ns/openshift-etcd...
2026-07-09T18:31:54.264358889Z Wrote inspect data to must-gather.
2026-07-09T18:31:54.264518163Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:31:54.264518163Z     namespaces "assisted-installer" not found
2026-07-09T18:31:54.772836209Z Waiting on subprocesses to finish execution.
2026-07-09T18:31:54.781331466Z INFO: Gathering HAProxy config files
2026-07-09T18:31:54.785657204Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T18:31:54.794998654Z Executing Istio gather script
2026-07-09T18:31:54.822409317Z INFO: Collecting host service logs for crio
2026-07-09T18:31:54.822799443Z INFO: Collecting host service logs for kubelet
2026-07-09T18:31:54.823209076Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T18:31:54.823529469Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T18:31:54.823830078Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T18:31:54.824102365Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T18:31:54.824415833Z INFO: Collecting host service logs for NetworkManager
2026-07-09T18:31:54.824706172Z INFO: Collecting host service logs for openvswitch
2026-07-09T18:31:54.824977618Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T18:31:54.825272293Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T18:31:54.825573642Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T18:31:54.825986046Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T18:31:54.842796778Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T18:31:54.843935849Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T18:31:55.043739107Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T18:31:55.344418880Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T18:31:55.708518540Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-g92nm
2026-07-09T18:31:55.712059167Z insights-runtime-extractor-hjwm6
2026-07-09T18:31:55.712114351Z insights-runtime-extractor-p7492
2026-07-09T18:31:55.712150399Z insights-runtime-extractor-q6mvh
2026-07-09T18:31:55.784019637Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T18:31:55.869991558Z No resources found
2026-07-09T18:31:56.085384144Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T18:31:56.096376371Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T18:31:56.096376371Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T18:31:56.099002020Z Wrote inspect data to must-gather.
2026-07-09T18:31:56.150660621Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T18:31:56.153639711Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T18:31:56.167374900Z ERROR: No running kube-apiserver pods found
2026-07-09T18:31:56.392538725Z No resources found in openshift-etcd namespace.
2026-07-09T18:31:56.399143814Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T18:31:56.469655993Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T18:31:56.469655993Z See 'oc rsync -h' for help and examples
2026-07-09T18:31:56.514795007Z INFO: Worker host service log collection to complete.
2026-07-09T18:31:56.659762786Z Wrote inspect data to must-gather.
2026-07-09T18:31:56.724080885Z Wrote inspect data to must-gather.
2026-07-09T18:31:56.857082752Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T18:31:56.889552112Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-09T18:31:56.933555190Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T18:31:57.163775344Z Wrote inspect data to must-gather.
2026-07-09T18:31:57.342657019Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:31:57.394804051Z INFO: 'previous-logs' folder not found on ip-10-0-129-181.ec2.internal, skipping...
2026-07-09T18:31:57.445971442Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T18:31:57.571034869Z No resources found
2026-07-09T18:31:57.612752893Z tar: Removing leading `/' from member names
2026-07-09T18:31:57.619082302Z INFO: HAProxy config collection complete.
2026-07-09T18:31:57.657287173Z Wrote inspect data to must-gather.
2026-07-09T18:31:57.720365223Z Gathering data for ns/openshift-monitoring...
2026-07-09T18:31:57.742951865Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T18:31:57.763153761Z Wrote inspect data to must-gather.
2026-07-09T18:31:57.795273985Z Wrote inspect data to must-gather.
2026-07-09T18:31:57.915536724Z INFO: 'previous-logs' folder not found on ip-10-0-142-201.ec2.internal, skipping...
2026-07-09T18:31:57.983841260Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T18:31:58.123050825Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T18:31:58.132771664Z Wrote inspect data to must-gather.
2026-07-09T18:31:58.137480640Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T18:31:58.255437985Z INFO: 'previous-logs' folder not found on ip-10-0-143-68.ec2.internal, skipping...
2026-07-09T18:31:58.255485463Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T18:31:58.255501901Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T18:31:58.288434016Z Wrote inspect data to must-gather.
2026-07-09T18:31:58.417286159Z Wrote inspect data to must-gather.
2026-07-09T18:31:58.471506397Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T18:31:58.599957098Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T18:31:58.693399004Z Wrote inspect data to must-gather.
2026-07-09T18:31:58.719040748Z Wrote inspect data to must-gather.
2026-07-09T18:31:58.954551860Z Wrote inspect data to must-gather.
2026-07-09T18:31:59.008756776Z INFO: INTERCONNECT MODE
2026-07-09T18:31:59.008790806Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T18:31:59.086581735Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T18:31:59.206305587Z INFO: Gathering OVN_Northbound from ovnkube-node-7fnv5...
2026-07-09T18:31:59.206554996Z INFO: Gathering OVN_Northbound from ovnkube-node-cqvlv...
2026-07-09T18:31:59.206816293Z INFO: Gathering OVN_Northbound from ovnkube-node-lbftk...
2026-07-09T18:31:59.207141526Z INFO: Gathering OVN_Southbound from ovnkube-node-7fnv5...
2026-07-09T18:31:59.208049438Z Wrote inspect data to must-gather.
2026-07-09T18:31:59.208286758Z INFO: Gathering OVN_Southbound from ovnkube-node-cqvlv...
2026-07-09T18:31:59.208585922Z INFO: Gathering OVN_Southbound from ovnkube-node-lbftk...
2026-07-09T18:31:59.405075645Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T18:31:59.556103764Z tar: Removing leading `/' from member names
2026-07-09T18:31:59.600188694Z tar: Removing leading `/' from member names
2026-07-09T18:31:59.636172724Z tar: Removing leading `/' from member names
2026-07-09T18:31:59.658961282Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T18:31:59.662937963Z tar: Removing leading `/' from member names
2026-07-09T18:31:59.664120423Z tar: Removing leading `/' from member names
2026-07-09T18:31:59.679170115Z tar: Removing leading `/' from member names
2026-07-09T18:31:59.686560591Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:31:59.704293195Z Wrote inspect data to must-gather.
2026-07-09T18:31:59.809163174Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T18:31:59.902121410Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:31:59.989679693Z Wrote inspect data to must-gather.
2026-07-09T18:32:00.134665835Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T18:32:00.145743807Z INFO: Waiting for network log collection to complete ...
2026-07-09T18:32:00.145879120Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T18:32:00.146527388Z INFO: Copying ovnk databases complete.
2026-07-09T18:32:00.148706397Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-09T18:32:00.156363022Z ovnk_database_store/
2026-07-09T18:32:00.156422527Z ovnk_database_store/ovnkube-node-7fnv5_nbdb
2026-07-09T18:32:00.166004681Z ovnk_database_store/ovnkube-node-cqvlv_nbdb
2026-07-09T18:32:00.176998625Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:00.180627161Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T18:32:00.180659011Z ovnk_database_store/ovnkube-node-7fnv5_sbdb
2026-07-09T18:32:00.202248262Z ovnk_database_store/ovnkube-node-lbftk_sbdb
2026-07-09T18:32:00.226198259Z ovnk_database_store/ovnkube-node-cqvlv_sbdb
2026-07-09T18:32:00.244921174Z ovnk_database_store/ovnkube-node-lbftk_nbdb
2026-07-09T18:32:00.250913525Z Wrote inspect data to must-gather.
2026-07-09T18:32:00.258560087Z INFO: Network log collection complete.
2026-07-09T18:32:00.391980079Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:32:00.415308454Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:00.606454337Z Wrote inspect data to must-gather.
2026-07-09T18:32:00.616578822Z INFO: Getting status from alertmanager-main-0
2026-07-09T18:32:00.627135381Z Gathering data for ns/openshift-network-console...
2026-07-09T18:32:00.676599818Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:00.679472720Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T18:32:00.904400841Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:00.910980110Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T18:32:00.913432187Z INFO: Fallback to identify the container image from release info
2026-07-09T18:32:01.141032049Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:01.144492391Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T18:32:01.198099821Z Gathering data for ns/openshift-console-operator...
2026-07-09T18:32:01.374357645Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:01.605399871Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:01.609913820Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-09T18:32:01.627393183Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-09T18:32:01.628327674Z Gathering data for ns/openshift-console...
2026-07-09T18:32:01.775834162Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T18:32:01.808584420Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:01.900021841Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-09T18:32:01.910684145Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T18:32:02.030650965Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:02.033139178Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-09T18:32:02.257471566Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:02.273567382Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T18:32:02.474359233Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:02.477355569Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-09T18:32:02.669656505Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:02.864001079Z Gathering data for ns/openshift-dns-operator...
2026-07-09T18:32:02.880167511Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:02.882712677Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T18:32:03.065128775Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T18:32:03.090680289Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:03.250742387Z Gathering data for ns/openshift-dns...
2026-07-09T18:32:03.305448928Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:03.308390605Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T18:32:03.463365134Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:03.659299393Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:03.661882825Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T18:32:03.843473641Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:03.963151349Z Gathering data for ns/openshift-image-registry...
2026-07-09T18:32:04.054383927Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:32:04.224268526Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T18:32:04.519718946Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T18:32:04.813077627Z Done executing Istio gather script
2026-07-09T18:32:04.927119043Z Gathering data for ns/openshift-ingress...
2026-07-09T18:32:05.300246018Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T18:32:05.363986880Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T18:32:05.749336155Z Gathering data for ns/openshift-insights...
2026-07-09T18:32:06.503420497Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T18:32:06.905189148Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-09T18:32:07.632196991Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T18:32:08.129396024Z Gathering data for ns/kserve...
2026-07-09T18:32:08.754801118Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T18:32:09.864503587Z Gathering data for ns/openshift-config...
2026-07-09T18:32:09.879709870Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T18:32:10.169036627Z Collecting performance related data for node ip-10-0-143-68.ec2.internal
2026-07-09T18:32:10.177182108Z Collecting performance related data for node ip-10-0-129-181.ec2.internal
2026-07-09T18:32:10.179770083Z Gathering data for ns/openshift-config-managed...
2026-07-09T18:32:10.186591562Z Collecting performance related data for node ip-10-0-142-201.ec2.internal
2026-07-09T18:32:10.827566956Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T18:32:11.220461237Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T18:32:11.586033454Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T18:32:11.912267257Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-09T18:32:11.997289537Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T18:32:12.350043803Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T18:32:12.659272184Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T18:32:13.056478912Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T18:32:13.534394146Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T18:32:13.895231777Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T18:32:14.538966992Z Gathering data for ns/openshift-multus...
2026-07-09T18:32:16.132089728Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T18:32:16.917336869Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-09T18:32:17.569839373Z Gathering data for ns/openshift-host-network...
2026-07-09T18:32:18.130160491Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T18:32:18.890937277Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T18:32:18.896540200Z Collecting kubelet logs for node ip-10-0-143-68.ec2.internal
2026-07-09T18:32:18.898196218Z Collecting kubelet logs for node ip-10-0-129-181.ec2.internal
2026-07-09T18:32:18.900403321Z Collecting kubelet logs for node ip-10-0-142-201.ec2.internal
2026-07-09T18:32:19.307048143Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-sbwk5 namespace
2026-07-09T18:32:19.312383756Z INFO: Node performance data collection complete.
2026-07-09T18:32:19.464472865Z Gathering data for ns/openshift-network-operator...
2026-07-09T18:32:19.915659775Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T18:32:20.221297939Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T18:32:20.773385738Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T18:32:21.232414781Z Gathering data for ns/openshift-apiserver...
2026-07-09T18:32:21.693668435Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T18:32:21.921983136Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-09T18:32:21.986601663Z Gathering data for ns/openshift-controller-manager...
2026-07-09T18:32:22.291605133Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T18:32:23.010872052Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T18:32:23.384755297Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T18:32:23.764240919Z Gathering data for ns/openshift-service-ca...
2026-07-09T18:32:24.207299621Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T18:32:25.269008790Z Wrote inspect data to must-gather.
2026-07-09T18:32:25.269057667Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:32:25.269057667Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T18:32:25.413845895Z error: the server doesn't have a resource type "clusters"
2026-07-09T18:32:25.538954732Z Caches written to disk