2026-10-30T04:18:11.764264346Z [disk usage checker] Started
2026-10-30T04:18:11.767614368Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:12.132705485Z Gathering data for ns/openshift-cluster-version...
2026-10-30T04:18:12.401013044Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-30T04:18:12.598109156Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-30T04:18:12.791344647Z Gathering data for ns/default...
2026-10-30T04:18:13.466724549Z Gathering data for ns/openshift...
2026-10-30T04:18:14.258648660Z Gathering data for ns/kube-system...
2026-10-30T04:18:15.271194411Z Waiting on subprocesses to finish execution.
2026-10-30T04:18:15.275676973Z INFO: Gathering HAProxy config files
2026-10-30T04:18:15.276861057Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-10-30T04:18:15.284438661Z INFO: Gathering machine config daemon's old logs from all nodes
2026-10-30T04:18:15.288481944Z Executing Istio gather script
2026-10-30T04:18:15.301506214Z INFO: Collecting host service logs for crio
2026-10-30T04:18:15.301815608Z INFO: Collecting host service logs for kubelet
2026-10-30T04:18:15.302112755Z INFO: Collecting host service logs for rpm-ostreed
2026-10-30T04:18:15.302437218Z INFO: Collecting host service logs for ostree-finalize-staged
2026-10-30T04:18:15.302745087Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-10-30T04:18:15.303054597Z INFO: Collecting host service logs for machine-config-daemon-host
2026-10-30T04:18:15.303379015Z INFO: Collecting host service logs for NetworkManager
2026-10-30T04:18:15.303684464Z INFO: Collecting host service logs for openvswitch
2026-10-30T04:18:15.303993647Z INFO: Collecting host service logs for ovs-configuration
2026-10-30T04:18:15.304320387Z INFO: Collecting host service logs for ovsdb-server
2026-10-30T04:18:15.304625986Z INFO: Collecting host service logs for ovs-vswitchd
2026-10-30T04:18:15.304975419Z INFO: Waiting for worker host service log collection to complete ...
2026-10-30T04:18:15.311987608Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-10-30T04:18:15.431462990Z INFO: Waiting for node performance related collection to complete ...
2026-10-30T04:18:15.915562579Z INFO: "sriov-network-operator" not detected. Skipping.
2026-10-30T04:18:16.035512724Z No resources found in openshift-etcd namespace.
2026-10-30T04:18:16.272037158Z Wrote inspect data to must-gather.
2026-10-30T04:18:16.400073868Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-10-30T04:18:16.474131341Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-gxb72
2026-10-30T04:18:16.474131341Z insights-runtime-extractor-bvj8c
2026-10-30T04:18:16.474131341Z insights-runtime-extractor-dkfnj
2026-10-30T04:18:16.474131341Z insights-runtime-extractor-xgrln
2026-10-30T04:18:16.510300422Z error: the server doesn't have a resource type "performanceprofile"
2026-10-30T04:18:16.674100424Z INFO: "nfd" not detected. Skipping.
2026-10-30T04:18:16.681245162Z INFO: Found 1 replicas - prometheus-k8s-0
2026-10-30T04:18:16.876295742Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:16.931942773Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-10-30T04:18:17.094351070Z INFO: "metallb-operator" not detected. Skipping.
2026-10-30T04:18:17.098996552Z No resources found
2026-10-30T04:18:17.106375882Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-10-30T04:18:17.106450238Z INFO: on-disk MachineConfig config collection complete.
2026-10-30T04:18:17.179189911Z INFO: Worker host service log collection to complete.
2026-10-30T04:18:17.463736958Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-30T04:18:17.515226901Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-10-30T04:18:17.515226901Z See 'oc rsync -h' for help and examples
2026-10-30T04:18:17.769016759Z INFO: Waiting for HAProxy config collection to complete ...
2026-10-30T04:18:17.835789351Z ERROR: No running kube-apiserver pods found
2026-10-30T04:18:17.879026508Z Gathering data for ns/openshift-etcd...
2026-10-30T04:18:17.969951857Z No resources found
2026-10-30T04:18:17.988447710Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-10-30T04:18:18.068323847Z Wrote inspect data to must-gather.
2026-10-30T04:18:18.235700872Z Wrote inspect data to must-gather.
2026-10-30T04:18:18.517152506Z INFO: 'previous-logs' folder not found on ip-10-0-128-202.ec2.internal, skipping...
2026-10-30T04:18:18.592610485Z Wrote inspect data to must-gather.
2026-10-30T04:18:18.598648226Z Wrote inspect data to must-gather.
2026-10-30T04:18:18.614431840Z tar: Removing leading `/' from member names
2026-10-30T04:18:18.637035700Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-10-30T04:18:18.645668897Z INFO: HAProxy config collection complete.
2026-10-30T04:18:18.700630500Z Gathering data for ns/openshift-cloud-credential-operator...
2026-10-30T04:18:18.895225095Z error: the server doesn't have a resource type "machineconfigs"
2026-10-30T04:18:18.897614040Z Wrote inspect data to must-gather.
2026-10-30T04:18:18.978135217Z Wrote inspect data to must-gather.
2026-10-30T04:18:19.117025222Z INFO: 'previous-logs' folder not found on ip-10-0-128-241.ec2.internal, skipping...
2026-10-30T04:18:19.235384066Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-10-30T04:18:19.253525305Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-30T04:18:19.311924704Z error: the server doesn't have a resource type "machineconfigpools"
2026-10-30T04:18:19.383731802Z Wrote inspect data to must-gather.
2026-10-30T04:18:19.383761254Z error: inspection completed with the errors occurred while gathering data:
2026-10-30T04:18:19.383761254Z     namespaces "assisted-installer" not found
2026-10-30T04:18:19.384139138Z INFO: Getting rules from prometheus-k8s-0
2026-10-30T04:18:19.616640684Z Wrote inspect data to must-gather.
2026-10-30T04:18:19.617258534Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-30T04:18:19.669888293Z INFO: 'previous-logs' folder not found on ip-10-0-133-197.ec2.internal, skipping...
2026-10-30T04:18:19.670018942Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-10-30T04:18:19.670018942Z INFO: Machine Config Daemon termination log collection complete.
2026-10-30T04:18:19.731420639Z Wrote inspect data to must-gather.
2026-10-30T04:18:19.981024531Z error: the server doesn't have a resource type "kubeletconfigs"
2026-10-30T04:18:20.104671783Z Wrote inspect data to must-gather.
2026-10-30T04:18:20.245991846Z INFO: Getting status/config from prometheus-k8s-0
2026-10-30T04:18:20.410769840Z Wrote inspect data to must-gather.
2026-10-30T04:18:20.563300767Z Gathering data for ns/openshift-monitoring...
2026-10-30T04:18:20.601422218Z INFO: LEGACY MODE
2026-10-30T04:18:20.601477464Z INFO: Gathering ovn-kubernetes master data
2026-10-30T04:18:20.724656126Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-10-30T04:18:20.787009904Z Wrote inspect data to must-gather.
2026-10-30T04:18:21.024601809Z INFO: Getting status/flags from prometheus-k8s-0
2026-10-30T04:18:21.062565649Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:21.152692348Z Wrote inspect data to must-gather.
2026-10-30T04:18:21.415610599Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-10-30T04:18:21.477073616Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:21.480772052Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-10-30T04:18:21.567695263Z Wrote inspect data to must-gather.
2026-10-30T04:18:21.800474574Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:21.853064633Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-10-30T04:18:21.887368891Z INFO: Waiting for network log collection to complete ...
2026-10-30T04:18:21.887536584Z INFO: Waiting for ovnk database copies to complete ...
2026-10-30T04:18:21.888116291Z INFO: Copying ovnk databases complete.
2026-10-30T04:18:21.892167284Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-10-30T04:18:21.892943435Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 704 is not a child of this shell
2026-10-30T04:18:21.892990325Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 705 is not a child of this shell
2026-10-30T04:18:21.893015302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 706 is not a child of this shell
2026-10-30T04:18:21.893037623Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 707 is not a child of this shell
2026-10-30T04:18:21.893060240Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 708 is not a child of this shell
2026-10-30T04:18:21.893082899Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 709 is not a child of this shell
2026-10-30T04:18:21.893104953Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 711 is not a child of this shell
2026-10-30T04:18:21.893127631Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 712 is not a child of this shell
2026-10-30T04:18:21.893150188Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 715 is not a child of this shell
2026-10-30T04:18:21.893172501Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 716 is not a child of this shell
2026-10-30T04:18:21.893194800Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 718 is not a child of this shell
2026-10-30T04:18:21.893217252Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 719 is not a child of this shell
2026-10-30T04:18:21.893239236Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 721 is not a child of this shell
2026-10-30T04:18:21.896999689Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-10-30T04:18:21.909445457Z tar: Exiting with failure status due to previous errors
2026-10-30T04:18:21.910125363Z INFO: Network log collection complete.
2026-10-30T04:18:21.924268154Z Wrote inspect data to must-gather.
2026-10-30T04:18:21.990556955Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:22.151615926Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:22.154696549Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-10-30T04:18:22.249957690Z Wrote inspect data to must-gather.
2026-10-30T04:18:22.334917827Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-10-30T04:18:22.450797410Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:22.540115378Z Wrote inspect data to must-gather.
2026-10-30T04:18:22.828666304Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:22.831582052Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-10-30T04:18:22.903336764Z INFO: Getting status from alertmanager-main-0
2026-10-30T04:18:22.962993898Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-10-30T04:18:23.129164559Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:23.177788976Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-10-30T04:18:23.366909462Z daemonset.apps/perf-node-gather-daemonset created
2026-10-30T04:18:23.487386345Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:23.490389946Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-10-30T04:18:23.533703219Z Waiting for performance profile collector pods to become ready: 1
2026-10-30T04:18:23.784733702Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:24.120232626Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:24.124042183Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-10-30T04:18:24.338084409Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:24.634672747Z Wrote inspect data to /must-gather/istio.
2026-10-30T04:18:24.696848111Z Waiting for performance profile collector pods to become ready: 2
2026-10-30T04:18:24.836437117Z Wrote inspect data to must-gather.
2026-10-30T04:18:25.641661680Z Done executing Istio gather script
2026-10-30T04:18:25.832927466Z Waiting for performance profile collector pods to become ready: 3
2026-10-30T04:18:26.974874591Z Waiting for performance profile collector pods to become ready: 4
2026-10-30T04:18:26.998332741Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:27.189690516Z Gathering data for ns/openshift-network-console...
2026-10-30T04:18:28.116353525Z Waiting for performance profile collector pods to become ready: 5
2026-10-30T04:18:28.152393042Z Gathering data for ns/openshift-console-operator...
2026-10-30T04:18:28.989126846Z Gathering data for ns/openshift-console...
2026-10-30T04:18:29.252145321Z Waiting for performance profile collector pods to become ready: 6
2026-10-30T04:18:30.194247507Z Gathering data for ns/openshift-cluster-storage-operator...
2026-10-30T04:18:30.389108263Z Waiting for performance profile collector pods to become ready: 7
2026-10-30T04:18:31.330228754Z Gathering data for ns/openshift-dns-operator...
2026-10-30T04:18:31.538849382Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-10-30T04:18:31.832954331Z Collecting performance related data for node ip-10-0-128-202.ec2.internal
2026-10-30T04:18:31.842974812Z Collecting performance related data for node ip-10-0-128-241.ec2.internal
2026-10-30T04:18:31.850357040Z Collecting performance related data for node ip-10-0-133-197.ec2.internal
2026-10-30T04:18:32.003104503Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:32.177611712Z Gathering data for ns/openshift-dns...
2026-10-30T04:18:34.135350755Z Gathering data for ns/openshift-image-registry...
2026-10-30T04:18:35.550545122Z Gathering data for ns/openshift-ingress-operator...
2026-10-30T04:18:36.298861187Z Gathering data for ns/openshift-ingress...
2026-10-30T04:18:37.007919369Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:37.146759282Z Gathering data for ns/openshift-ingress-canary...
2026-10-30T04:18:38.241955655Z Gathering data for ns/openshift-insights...
2026-10-30T04:18:39.697266898Z Collecting kubelet logs for node ip-10-0-128-202.ec2.internal
2026-10-30T04:18:39.699073494Z Collecting kubelet logs for node ip-10-0-128-241.ec2.internal
2026-10-30T04:18:39.700866786Z Collecting kubelet logs for node ip-10-0-133-197.ec2.internal
2026-10-30T04:18:40.309449196Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-6xn7p namespace
2026-10-30T04:18:40.328668546Z INFO: Node performance data collection complete.
2026-10-30T04:18:41.369525556Z Gathering data for ns/openshift-operators-redhat...
2026-10-30T04:18:42.012528035Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:43.070751036Z Gathering data for ns/openshift-opentelemetry-operator...
2026-10-30T04:18:44.284615334Z Gathering data for ns/openshift-cluster-observability-operator...
2026-10-30T04:18:46.325972790Z Gathering data for ns/openshift-config...
2026-10-30T04:18:46.984568429Z Gathering data for ns/openshift-config-managed...
2026-10-30T04:18:47.017273974Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:47.792217137Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-10-30T04:18:48.446423496Z Gathering data for ns/openshift-kube-apiserver...
2026-10-30T04:18:49.100078611Z Gathering data for ns/openshift-kube-controller-manager...
2026-10-30T04:18:49.811187230Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-10-30T04:18:50.519502317Z Gathering data for ns/openshift-kube-scheduler...
2026-10-30T04:18:51.173127570Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-10-30T04:18:51.885258371Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-10-30T04:18:52.022147482Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:52.798374351Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-10-30T04:18:53.619380868Z Gathering data for ns/openshift-user-workload-monitoring...
2026-10-30T04:18:54.702512588Z Gathering data for ns/openshift-multus...
2026-10-30T04:18:57.026994832Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:18:59.818178837Z Gathering data for ns/openshift-ovn-kubernetes...
2026-10-30T04:19:02.031615781Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:19:04.106162607Z Gathering data for ns/openshift-host-network...
2026-10-30T04:19:04.883019963Z Gathering data for ns/openshift-network-diagnostics...
2026-10-30T04:19:06.444881627Z Gathering data for ns/openshift-network-node-identity...
2026-10-30T04:19:07.036309184Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:19:07.553050910Z Gathering data for ns/openshift-network-operator...
2026-10-30T04:19:08.607622585Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-10-30T04:19:09.261988389Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-10-30T04:19:10.491947666Z Gathering data for ns/openshift-apiserver-operator...
2026-10-30T04:19:11.186236199Z Gathering data for ns/openshift-apiserver...
2026-10-30T04:19:12.041767832Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:19:12.129018962Z Gathering data for ns/openshift-controller-manager-operator...
2026-10-30T04:19:12.786526637Z Gathering data for ns/openshift-controller-manager...
2026-10-30T04:19:13.495722076Z Gathering data for ns/openshift-cluster-samples-operator...
2026-10-30T04:19:14.736107486Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-10-30T04:19:15.428880640Z Gathering data for ns/openshift-service-ca-operator...
2026-10-30T04:19:16.229920775Z Gathering data for ns/openshift-service-ca...
2026-10-30T04:19:17.047095883Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T04:19:17.171520101Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-10-30T04:19:19.348357380Z Wrote inspect data to must-gather.
2026-10-30T04:19:19.348404489Z error: inspection completed with the errors occurred while gathering data:
2026-10-30T04:19:19.348404489Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-10-30T04:19:19.550038846Z error: the server doesn't have a resource type "clusters"
2026-10-30T04:19:19.653506870Z Caches written to disk