2026-07-01T18:34:29.055410754Z [disk usage checker] Started 2026-07-01T18:34:29.058727861Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-01T18:34:29.261312706Z Gathering data for ns/openshift-cluster-version... 2026-07-01T18:34:29.409787117Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-01T18:34:29.501764347Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-01T18:34:29.582641938Z Gathering data for ns/default... 2026-07-01T18:34:29.914437034Z Gathering data for ns/openshift... 2026-07-01T18:34:30.325126326Z Gathering data for ns/kube-system... 2026-07-01T18:34:31.075195271Z Gathering data for ns/openshift-etcd... 2026-07-01T18:34:31.409020967Z Wrote inspect data to must-gather. 2026-07-01T18:34:31.409074466Z error: inspection completed with the errors occurred while gathering data: 2026-07-01T18:34:31.409074466Z namespaces "assisted-installer" not found 2026-07-01T18:34:31.818786645Z Waiting on subprocesses to finish execution. 2026-07-01T18:34:31.822391514Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-01T18:34:31.827610796Z Executing Istio gather script 2026-07-01T18:34:31.830135084Z INFO: Gathering HAProxy config files 2026-07-01T18:34:31.841944457Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-01T18:34:31.842524121Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-01T18:34:31.853657112Z INFO: Waiting for node performance related collection to complete ... 2026-07-01T18:34:31.864523411Z INFO: Collecting host service logs for crio 2026-07-01T18:34:31.864810491Z INFO: Collecting host service logs for kubelet 2026-07-01T18:34:31.865136855Z INFO: Collecting host service logs for rpm-ostreed 2026-07-01T18:34:31.865415484Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-01T18:34:31.865710060Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-01T18:34:31.865998676Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-01T18:34:31.866313163Z INFO: Collecting host service logs for NetworkManager 2026-07-01T18:34:31.866618882Z INFO: Collecting host service logs for openvswitch 2026-07-01T18:34:31.866942702Z INFO: Collecting host service logs for ovs-configuration 2026-07-01T18:34:31.867262955Z INFO: Collecting host service logs for ovsdb-server 2026-07-01T18:34:31.867566960Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-01T18:34:31.867930173Z INFO: Waiting for worker host service log collection to complete ... 2026-07-01T18:34:32.376581230Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-s8mwl 2026-07-01T18:34:32.376581230Z insights-runtime-extractor-92l4g 2026-07-01T18:34:32.376581230Z insights-runtime-extractor-j2chw 2026-07-01T18:34:32.376581230Z insights-runtime-extractor-lfktk 2026-07-01T18:34:32.433321901Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-01T18:34:32.433365828Z INFO: on-disk MachineConfig config collection complete. 2026-07-01T18:34:32.672009115Z error: the server doesn't have a resource type "performanceprofile" 2026-07-01T18:34:32.731941695Z No resources found in openshift-etcd namespace. 2026-07-01T18:34:32.902459172Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-01T18:34:32.910017373Z No resources found 2026-07-01T18:34:32.994773123Z INFO: "metallb-operator" not detected. Skipping. 2026-07-01T18:34:33.022377514Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-01T18:34:33.219991863Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-01T18:34:33.273594036Z INFO: Worker host service log collection to complete. 2026-07-01T18:34:33.314101266Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-01T18:34:33.314101266Z See 'oc rsync -h' for help and examples 2026-07-01T18:34:33.431498640Z ERROR: No running kube-apiserver pods found 2026-07-01T18:34:33.571170832Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-01T18:34:33.621875757Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-01T18:34:33.740049855Z Wrote inspect data to must-gather. 2026-07-01T18:34:33.875876680Z Wrote inspect data to must-gather. 2026-07-01T18:34:33.879474595Z Gathering data for ns/openshift-monitoring... 2026-07-01T18:34:33.996596691Z No resources found 2026-07-01T18:34:34.094009886Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-01T18:34:34.118296793Z Wrote inspect data to must-gather. 2026-07-01T18:34:34.190288664Z Wrote inspect data to must-gather. 2026-07-01T18:34:34.272590981Z Wrote inspect data to must-gather. 2026-07-01T18:34:34.286849306Z Wrote inspect data to must-gather. 2026-07-01T18:34:34.298478109Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-01T18:34:34.359542324Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-01T18:34:34.372253778Z Wrote inspect data to must-gather. 2026-07-01T18:34:34.388840884Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-01T18:34:34.417337369Z INFO: 'previous-logs' folder not found on ip-10-0-136-158.ec2.internal, skipping... 2026-07-01T18:34:34.597224007Z error: the server doesn't have a resource type "machineconfigs" 2026-07-01T18:34:34.633845654Z INFO: HAProxy config collection complete. 2026-07-01T18:34:34.801868106Z Wrote inspect data to must-gather. 2026-07-01T18:34:34.827938291Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-01T18:34:34.829203535Z INFO: 'previous-logs' folder not found on ip-10-0-139-128.ec2.internal, skipping... 2026-07-01T18:34:34.995050380Z INFO: Getting rules from prometheus-k8s-0 2026-07-01T18:34:35.098245106Z Wrote inspect data to must-gather. 2026-07-01T18:34:35.107333649Z Wrote inspect data to must-gather. 2026-07-01T18:34:35.208683351Z INFO: 'previous-logs' folder not found on ip-10-0-140-183.ec2.internal, skipping... 2026-07-01T18:34:35.208782941Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-01T18:34:35.208812049Z INFO: Machine Config Daemon termination log collection complete. 2026-07-01T18:34:35.270836986Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-01T18:34:35.280293957Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-01T18:34:35.381576538Z Wrote inspect data to must-gather. 2026-07-01T18:34:35.464462361Z INFO: Getting status/config from prometheus-k8s-0 2026-07-01T18:34:35.538635212Z Wrote inspect data to must-gather. 2026-07-01T18:34:35.749368085Z INFO: INTERCONNECT MODE 2026-07-01T18:34:35.749400207Z INFO: Gathering ovn-kubernetes DBs 2026-07-01T18:34:35.773628632Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-01T18:34:35.784551097Z Wrote inspect data to must-gather. 2026-07-01T18:34:35.938207960Z INFO: Gathering OVN_Northbound from ovnkube-node-fmhdl... 2026-07-01T18:34:35.938407209Z INFO: Gathering OVN_Northbound from ovnkube-node-khzgt... 2026-07-01T18:34:35.938619712Z INFO: Gathering OVN_Northbound from ovnkube-node-xrzqw... 2026-07-01T18:34:35.938903380Z INFO: Gathering OVN_Southbound from ovnkube-node-fmhdl... 2026-07-01T18:34:35.939158962Z INFO: Gathering OVN_Southbound from ovnkube-node-khzgt... 2026-07-01T18:34:35.939439592Z INFO: Gathering OVN_Southbound from ovnkube-node-xrzqw... 2026-07-01T18:34:36.030669291Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-01T18:34:36.080764436Z Wrote inspect data to must-gather. 2026-07-01T18:34:36.242570319Z tar: Removing leading `/' from member names 2026-07-01T18:34:36.259332201Z tar: Removing leading `/' from member names 2026-07-01T18:34:36.284804723Z tar: Removing leading `/' from member names 2026-07-01T18:34:36.300617147Z tar: Removing leading `/' from member names 2026-07-01T18:34:36.301381670Z tar: Removing leading `/' from member names 2026-07-01T18:34:36.322127929Z tar: Removing leading `/' from member names 2026-07-01T18:34:36.405204070Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-01T18:34:36.412705515Z Wrote inspect data to must-gather. 2026-07-01T18:34:36.568019345Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-01T18:34:36.624529374Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-01T18:34:36.674438240Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-01T18:34:36.778950952Z Wrote inspect data to must-gather. 2026-07-01T18:34:36.802312034Z Gathering data for ns/openshift-network-console... 2026-07-01T18:34:36.862079603Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:36.957202509Z INFO: Waiting for network log collection to complete ... 2026-07-01T18:34:36.957320188Z INFO: Waiting for ovnk database copies to complete ... 2026-07-01T18:34:36.957827129Z INFO: Copying ovnk databases complete. 2026-07-01T18:34:36.958010636Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-01T18:34:36.959990505Z 3.3M must-gather/network_logs/ovnk_database_store 2026-07-01T18:34:36.962479754Z ovnk_database_store/ 2026-07-01T18:34:36.962479754Z ovnk_database_store/ovnkube-node-xrzqw_sbdb 2026-07-01T18:34:36.985306591Z ovnk_database_store/ovnkube-node-xrzqw_nbdb 2026-07-01T18:34:36.994307452Z ovnk_database_store/ovnkube-node-khzgt_sbdb 2026-07-01T18:34:37.012907168Z ovnk_database_store/ovnkube-node-fmhdl_sbdb 2026-07-01T18:34:37.038679796Z ovnk_database_store/ovnkube-node-fmhdl_nbdb 2026-07-01T18:34:37.048777010Z ovnk_database_store/ovnkube-node-khzgt_nbdb 2026-07-01T18:34:37.063105355Z INFO: Network log collection complete. 2026-07-01T18:34:37.143117350Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:37.145555375Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-01T18:34:37.158164427Z Wrote inspect data to must-gather. 2026-07-01T18:34:37.337323232Z Gathering data for ns/openshift-console-operator... 2026-07-01T18:34:37.384371270Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:37.398394571Z INFO: Getting status from alertmanager-main-0 2026-07-01T18:34:37.503752440Z Wrote inspect data to must-gather. 2026-07-01T18:34:37.643676969Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:37.648888515Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-01T18:34:37.815628147Z Gathering data for ns/openshift-console... 2026-07-01T18:34:37.822238660Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-01T18:34:37.824607931Z INFO: Fallback to identify the container image from release info 2026-07-01T18:34:37.858495920Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:38.064940249Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:38.067568026Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-01T18:34:38.297044702Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:38.348659023Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-01T18:34:38.511782081Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:38.514086414Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-01T18:34:38.531490762Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f 2026-07-01T18:34:38.695662995Z daemonset.apps/perf-node-gather-daemonset created 2026-07-01T18:34:38.729564266Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:38.851494826Z Waiting for performance profile collector pods to become ready: 1 2026-07-01T18:34:38.955611750Z Gathering data for ns/openshift-dns-operator... 2026-07-01T18:34:38.963772933Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:38.966851665Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-01T18:34:39.099552872Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-01T18:34:39.156841916Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:39.334594204Z Gathering data for ns/openshift-dns... 2026-07-01T18:34:39.378140480Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:39.380528212Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-01T18:34:39.539757880Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:39.732502819Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:39.735803474Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-01T18:34:39.927460390Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:40.004212270Z Gathering data for ns/openshift-image-registry... 2026-07-01T18:34:40.011283834Z Waiting for performance profile collector pods to become ready: 2 2026-07-01T18:34:40.121162784Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:40.124173693Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-01T18:34:40.296580126Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:40.498637952Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:40.501711142Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-01T18:34:40.507847702Z Gathering data for ns/openshift-ingress-operator... 2026-07-01T18:34:40.669949607Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:40.858742894Z Gathering data for ns/openshift-ingress... 2026-07-01T18:34:40.873727944Z Wrote inspect data to /must-gather/istio. 2026-07-01T18:34:41.149954270Z Waiting for performance profile collector pods to become ready: 3 2026-07-01T18:34:41.226609644Z Gathering data for ns/openshift-ingress-canary... 2026-07-01T18:34:41.590993063Z Done executing Istio gather script 2026-07-01T18:34:41.635118978Z Gathering data for ns/openshift-insights... 2026-07-01T18:34:42.283906624Z Waiting for performance profile collector pods to become ready: 4 2026-07-01T18:34:43.419878477Z Waiting for performance profile collector pods to become ready: 5 2026-07-01T18:34:43.839651518Z Gathering data for ns/kserve... 2026-07-01T18:34:44.104297821Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-01T18:34:44.567745001Z Waiting for performance profile collector pods to become ready: 6 2026-07-01T18:34:45.496485645Z Gathering data for ns/openshift-config... 2026-07-01T18:34:45.690384242Z Waiting for performance profile collector pods to become ready: 7 2026-07-01T18:34:45.767162333Z Gathering data for ns/openshift-config-managed... 2026-07-01T18:34:46.185735848Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-01T18:34:46.440496565Z Gathering data for ns/openshift-kube-apiserver... 2026-07-01T18:34:46.696511207Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-01T18:34:46.821441222Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-01T18:34:47.044324443Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-01T18:34:47.096972833Z Collecting performance related data for node ip-10-0-139-128.ec2.internal 2026-07-01T18:34:47.106895439Z Collecting performance related data for node ip-10-0-136-158.ec2.internal 2026-07-01T18:34:47.120109993Z Collecting performance related data for node ip-10-0-140-183.ec2.internal 2026-07-01T18:34:47.391235886Z Gathering data for ns/openshift-kube-scheduler... 2026-07-01T18:34:47.752294948Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-01T18:34:48.112904064Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-01T18:34:48.506728193Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-01T18:34:48.863135093Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-01T18:34:49.109700347Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-01T18:34:49.417670398Z Gathering data for ns/openshift-multus... 2026-07-01T18:34:50.791589991Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-01T18:34:52.067159027Z Gathering data for ns/openshift-host-network... 2026-07-01T18:34:52.423944230Z Gathering data for ns/openshift-network-diagnostics... 2026-07-01T18:34:52.984965374Z Gathering data for ns/openshift-network-node-identity... 2026-07-01T18:34:53.471754367Z Gathering data for ns/openshift-network-operator... 2026-07-01T18:34:53.850381976Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-01T18:34:54.116175254Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-07-01T18:34:54.135258055Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-01T18:34:54.625013916Z Gathering data for ns/openshift-apiserver-operator... 2026-07-01T18:34:54.920821524Z Gathering data for ns/openshift-apiserver... 2026-07-01T18:34:55.321305683Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-01T18:34:55.585513200Z Gathering data for ns/openshift-controller-manager... 2026-07-01T18:34:55.598379279Z Collecting kubelet logs for node ip-10-0-139-128.ec2.internal 2026-07-01T18:34:55.600906028Z Collecting kubelet logs for node ip-10-0-136-158.ec2.internal 2026-07-01T18:34:55.603122007Z Collecting kubelet logs for node ip-10-0-140-183.ec2.internal 2026-07-01T18:34:55.967408301Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-01T18:34:55.979994559Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-9vfjm namespace 2026-07-01T18:34:55.984966811Z INFO: Node performance data collection complete. 2026-07-01T18:34:56.638961997Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-01T18:34:56.942539912Z Gathering data for ns/openshift-service-ca-operator... 2026-07-01T18:34:57.246163999Z Gathering data for ns/openshift-service-ca... 2026-07-01T18:34:57.612038209Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-01T18:34:58.512203762Z Wrote inspect data to must-gather. 2026-07-01T18:34:58.512252943Z error: inspection completed with the errors occurred while gathering data: 2026-07-01T18:34:58.512252943Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-01T18:34:58.653167141Z error: the server doesn't have a resource type "clusters" 2026-07-01T18:34:58.797430445Z Caches written to disk