2026-07-28T12:39:19.790387037Z [disk usage checker] Started
2026-07-28T12:39:19.794197101Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T12:39:19.981533488Z Gathering data for ns/openshift-cluster-version...
2026-07-28T12:39:20.102622485Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T12:39:20.187394756Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T12:39:20.262466431Z Gathering data for ns/default...
2026-07-28T12:39:20.606621940Z Gathering data for ns/openshift...
2026-07-28T12:39:21.016214659Z Gathering data for ns/kube-system...
2026-07-28T12:39:21.736854220Z Gathering data for ns/openshift-etcd...
2026-07-28T12:39:22.037282192Z Wrote inspect data to must-gather.
2026-07-28T12:39:22.037368135Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T12:39:22.037368135Z     namespaces "assisted-installer" not found
2026-07-28T12:39:22.287518781Z Waiting on subprocesses to finish execution.
2026-07-28T12:39:22.296246932Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T12:39:22.297090909Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T12:39:22.297118116Z INFO: Collecting host service logs for crio
2026-07-28T12:39:22.297395651Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T12:39:22.297395651Z INFO: Collecting host service logs for kubelet
2026-07-28T12:39:22.297719033Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T12:39:22.298063514Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T12:39:22.298406346Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T12:39:22.298842797Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T12:39:22.299177352Z INFO: Collecting host service logs for NetworkManager
2026-07-28T12:39:22.299504970Z INFO: Collecting host service logs for openvswitch
2026-07-28T12:39:22.299862153Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T12:39:22.300199178Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T12:39:22.300235633Z Executing Istio gather script
2026-07-28T12:39:22.300741923Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T12:39:22.301152775Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T12:39:22.310066523Z INFO: Gathering HAProxy config files
2026-07-28T12:39:22.332831820Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T12:39:23.006360435Z Wrote inspect data to must-gather.
2026-07-28T12:39:23.287064427Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T12:39:23.299288496Z No resources found in openshift-etcd namespace.
2026-07-28T12:39:23.337430336Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T12:39:23.385237048Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T12:39:23.436067097Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T12:39:23.438141977Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T12:39:23.486940206Z No resources found
2026-07-28T12:39:23.494471159Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T12:39:23.494504611Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T12:39:23.740748946Z ERROR: No running kube-apiserver pods found
2026-07-28T12:39:23.778327222Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T12:39:23.852466224Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-rrg9b
2026-07-28T12:39:23.852527362Z insights-runtime-extractor-7sr4g
2026-07-28T12:39:23.852550302Z insights-runtime-extractor-lxb9z
2026-07-28T12:39:23.852573489Z insights-runtime-extractor-qm68g
2026-07-28T12:39:23.867362739Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T12:39:23.920086831Z INFO: Worker host service log collection to complete.
2026-07-28T12:39:23.925313564Z Wrote inspect data to must-gather.
2026-07-28T12:39:23.968594132Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T12:39:24.407797161Z Wrote inspect data to must-gather.
2026-07-28T12:39:24.516442675Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T12:39:24.516442675Z See 'oc rsync -h' for help and examples
2026-07-28T12:39:24.544005464Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T12:39:24.554514568Z tar: Removing leading `/' from member names
2026-07-28T12:39:24.566515357Z INFO: HAProxy config collection complete.
2026-07-28T12:39:24.584235010Z No resources found
2026-07-28T12:39:24.626737146Z Wrote inspect data to must-gather.
2026-07-28T12:39:24.656950752Z Wrote inspect data to must-gather.
2026-07-28T12:39:24.660416464Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T12:39:24.670779081Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T12:39:24.730487725Z Wrote inspect data to must-gather.
2026-07-28T12:39:24.788925694Z Wrote inspect data to must-gather.
2026-07-28T12:39:24.796307883Z INFO: 'previous-logs' folder not found on ip-10-0-129-158.ec2.internal, skipping...
2026-07-28T12:39:24.803452489Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T12:39:24.847151491Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T12:39:24.972113192Z Wrote inspect data to must-gather.
2026-07-28T12:39:24.999242569Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T12:39:25.042294051Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T12:39:25.057504305Z Gathering data for ns/openshift-monitoring...
2026-07-28T12:39:25.111319352Z INFO: 'previous-logs' folder not found on ip-10-0-130-120.ec2.internal, skipping...
2026-07-28T12:39:25.245436846Z Wrote inspect data to must-gather.
2026-07-28T12:39:25.288114830Z Wrote inspect data to must-gather.
2026-07-28T12:39:25.398500307Z INFO: 'previous-logs' folder not found on ip-10-0-134-240.ec2.internal, skipping...
2026-07-28T12:39:25.398537091Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T12:39:25.398537091Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T12:39:25.462159352Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T12:39:25.474578815Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T12:39:25.549415909Z Wrote inspect data to must-gather.
2026-07-28T12:39:25.653461871Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T12:39:25.682705222Z Wrote inspect data to must-gather.
2026-07-28T12:39:25.827094235Z INFO: INTERCONNECT MODE
2026-07-28T12:39:25.827094235Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T12:39:25.850046832Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:25.850983444Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T12:39:25.908066988Z Wrote inspect data to must-gather.
2026-07-28T12:39:25.989293787Z INFO: Gathering OVN_Northbound from ovnkube-node-8zzvz...
2026-07-28T12:39:25.989554898Z INFO: Gathering OVN_Northbound from ovnkube-node-ksg7x...
2026-07-28T12:39:25.989842722Z INFO: Gathering OVN_Northbound from ovnkube-node-v95xv...
2026-07-28T12:39:25.990116868Z INFO: Gathering OVN_Southbound from ovnkube-node-8zzvz...
2026-07-28T12:39:25.990380897Z INFO: Gathering OVN_Southbound from ovnkube-node-ksg7x...
2026-07-28T12:39:25.990688698Z INFO: Gathering OVN_Southbound from ovnkube-node-v95xv...
2026-07-28T12:39:26.122261706Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T12:39:26.137182603Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:26.143082096Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T12:39:26.210264594Z Wrote inspect data to must-gather.
2026-07-28T12:39:26.269717515Z tar: Removing leading `/' from member names
2026-07-28T12:39:26.300799340Z tar: Removing leading `/' from member names
2026-07-28T12:39:26.321061277Z tar: Removing leading `/' from member names
2026-07-28T12:39:26.363144439Z tar: Removing leading `/' from member names
2026-07-28T12:39:26.384142683Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:26.426527400Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T12:39:26.431914217Z tar: Removing leading `/' from member names
2026-07-28T12:39:26.446339448Z Wrote inspect data to must-gather.
2026-07-28T12:39:26.467073670Z tar: Removing leading `/' from member names
2026-07-28T12:39:26.626132272Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:26.631761210Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T12:39:26.685217392Z Wrote inspect data to must-gather.
2026-07-28T12:39:26.694789811Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T12:39:26.857789361Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:26.861206347Z INFO: Waiting for network log collection to complete ...
2026-07-28T12:39:26.861238686Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T12:39:26.861625138Z INFO: Copying ovnk databases complete.
2026-07-28T12:39:26.863294597Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-28T12:39:26.865833701Z ovnk_database_store/
2026-07-28T12:39:26.865907165Z ovnk_database_store/ovnkube-node-8zzvz_sbdb
2026-07-28T12:39:26.888329744Z ovnk_database_store/ovnkube-node-v95xv_sbdb
2026-07-28T12:39:26.909274145Z ovnk_database_store/ovnkube-node-ksg7x_sbdb
2026-07-28T12:39:26.923632533Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T12:39:26.925161830Z Wrote inspect data to must-gather.
2026-07-28T12:39:26.931829353Z ovnk_database_store/ovnkube-node-v95xv_nbdb
2026-07-28T12:39:26.939855995Z ovnk_database_store/ovnkube-node-8zzvz_nbdb
2026-07-28T12:39:26.949189175Z ovnk_database_store/ovnkube-node-ksg7x_nbdb
2026-07-28T12:39:26.961119421Z INFO: Network log collection complete.
2026-07-28T12:39:27.066139368Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:27.069881746Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T12:39:27.101284556Z INFO: Getting status from alertmanager-main-0
2026-07-28T12:39:27.147956487Z Wrote inspect data to must-gather.
2026-07-28T12:39:27.297888522Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:27.398755204Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T12:39:27.401317538Z INFO: Fallback to identify the container image from release info
2026-07-28T12:39:27.496392780Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:27.498692597Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T12:39:27.521371871Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T12:39:27.648453694Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:27.686887274Z Gathering data for ns/openshift-network-console...
2026-07-28T12:39:27.832751953Z Wrote inspect data to /must-gather/istio.
2026-07-28T12:39:27.920149523Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T12:39:28.049879071Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T12:39:28.173548506Z Gathering data for ns/openshift-console-operator...
2026-07-28T12:39:28.182275715Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T12:39:28.518076549Z Done executing Istio gather script
2026-07-28T12:39:28.572827810Z Gathering data for ns/openshift-console...
2026-07-28T12:39:29.097381131Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T12:39:29.279798981Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T12:39:29.563380798Z Gathering data for ns/openshift-dns-operator...
2026-07-28T12:39:29.809874139Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T12:39:29.828135988Z Gathering data for ns/openshift-dns...
2026-07-28T12:39:30.399088738Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T12:39:30.399431877Z Gathering data for ns/openshift-image-registry...
2026-07-28T12:39:30.838182185Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T12:39:31.140541556Z Gathering data for ns/openshift-ingress...
2026-07-28T12:39:31.494692924Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T12:39:31.516955298Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T12:39:31.930463783Z Gathering data for ns/openshift-insights...
2026-07-28T12:39:32.636003761Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T12:39:33.739063738Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T12:39:33.942497255Z Gathering data for ns/openshift-config...
2026-07-28T12:39:34.203865647Z Gathering data for ns/openshift-config-managed...
2026-07-28T12:39:34.589514888Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T12:39:34.816325263Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T12:39:34.863618060Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T12:39:34.885576401Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T12:39:35.119727318Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T12:39:35.434137738Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T12:39:35.710465768Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T12:39:35.949780019Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T12:39:35.992348970Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T12:39:36.240555334Z Collecting performance related data for node ip-10-0-129-158.ec2.internal
2026-07-28T12:39:36.246592989Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T12:39:36.248448504Z Collecting performance related data for node ip-10-0-130-120.ec2.internal
2026-07-28T12:39:36.256170695Z Collecting performance related data for node ip-10-0-134-240.ec2.internal
2026-07-28T12:39:36.597921982Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T12:39:36.967404240Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T12:39:37.519194784Z Gathering data for ns/openshift-multus...
2026-07-28T12:39:38.991234711Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T12:39:39.821402692Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T12:39:40.301807915Z Gathering data for ns/openshift-host-network...
2026-07-28T12:39:40.668319639Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T12:39:41.189881930Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T12:39:41.616929098Z Gathering data for ns/openshift-network-operator...
2026-07-28T12:39:42.022450219Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T12:39:42.309831835Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T12:39:42.759001291Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T12:39:43.048579433Z Gathering data for ns/openshift-apiserver...
2026-07-28T12:39:43.460647119Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T12:39:43.721000143Z Gathering data for ns/openshift-controller-manager...
2026-07-28T12:39:43.991395607Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T12:39:44.187700783Z Collecting kubelet logs for node ip-10-0-129-158.ec2.internal
2026-07-28T12:39:44.189811399Z Collecting kubelet logs for node ip-10-0-130-120.ec2.internal
2026-07-28T12:39:44.192120488Z Collecting kubelet logs for node ip-10-0-134-240.ec2.internal
2026-07-28T12:39:44.539194669Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8gzv9 namespace
2026-07-28T12:39:44.543756267Z INFO: Node performance data collection complete.
2026-07-28T12:39:44.706589597Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T12:39:44.826186773Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T12:39:45.010539702Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T12:39:45.302744267Z Gathering data for ns/openshift-service-ca...
2026-07-28T12:39:45.636608085Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T12:39:46.453229380Z Wrote inspect data to must-gather.
2026-07-28T12:39:46.453274301Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T12:39:46.453274301Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T12:39:46.576553484Z error: the server doesn't have a resource type "clusters"
2026-07-28T12:39:46.707413604Z Caches written to disk