2026-07-23T14:19:34.935360980Z [disk usage checker] Started
2026-07-23T14:19:34.938874859Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-23T14:19:35.139986109Z Gathering data for ns/openshift-cluster-version...
2026-07-23T14:19:35.268171283Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-23T14:19:35.352736231Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-23T14:19:35.431553067Z Gathering data for ns/default...
2026-07-23T14:19:35.713703548Z Gathering data for ns/openshift...
2026-07-23T14:19:36.108233946Z Gathering data for ns/kube-system...
2026-07-23T14:19:36.784127919Z Gathering data for ns/openshift-etcd...
2026-07-23T14:19:37.082939517Z Wrote inspect data to must-gather.
2026-07-23T14:19:37.082978184Z error: inspection completed with the errors occurred while gathering data:
2026-07-23T14:19:37.082978184Z     namespaces "assisted-installer" not found
2026-07-23T14:19:37.512373992Z Waiting on subprocesses to finish execution.
2026-07-23T14:19:37.519224458Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-23T14:19:37.521144666Z INFO: Gathering HAProxy config files
2026-07-23T14:19:37.529688374Z Executing Istio gather script
2026-07-23T14:19:37.539836325Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-23T14:19:37.542528038Z INFO: Collecting host service logs for crio
2026-07-23T14:19:37.543036185Z INFO: Collecting host service logs for kubelet
2026-07-23T14:19:37.543413142Z INFO: Collecting host service logs for rpm-ostreed
2026-07-23T14:19:37.543730628Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-23T14:19:37.544056073Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-23T14:19:37.544385295Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-23T14:19:37.544684035Z INFO: Collecting host service logs for NetworkManager
2026-07-23T14:19:37.545056223Z INFO: Collecting host service logs for openvswitch
2026-07-23T14:19:37.545352969Z INFO: Collecting host service logs for ovs-configuration
2026-07-23T14:19:37.545666818Z INFO: Collecting host service logs for ovsdb-server
2026-07-23T14:19:37.546039160Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-23T14:19:37.546427569Z INFO: Waiting for worker host service log collection to complete ...
2026-07-23T14:19:37.559722128Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-23T14:19:37.586112345Z INFO: Waiting for node performance related collection to complete ...
2026-07-23T14:19:38.261289064Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-23T14:19:38.283327298Z No resources found
2026-07-23T14:19:38.410072567Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-23T14:19:38.410221587Z INFO: on-disk MachineConfig config collection complete.
2026-07-23T14:19:38.621521223Z No resources found in openshift-etcd namespace.
2026-07-23T14:19:38.693674328Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-kf559
2026-07-23T14:19:38.693737415Z insights-runtime-extractor-4xlxs
2026-07-23T14:19:38.693758204Z insights-runtime-extractor-k5s8t
2026-07-23T14:19:38.693777949Z insights-runtime-extractor-txckv
2026-07-23T14:19:38.701869338Z error: the server doesn't have a resource type "performanceprofile"
2026-07-23T14:19:38.767424584Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-23T14:19:38.835538951Z INFO: "metallb-operator" not detected. Skipping.
2026-07-23T14:19:38.873376366Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-23T14:19:38.978705518Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-23T14:19:38.978705518Z See 'oc rsync -h' for help and examples
2026-07-23T14:19:38.989254789Z Wrote inspect data to must-gather.
2026-07-23T14:19:39.001570203Z INFO: Worker host service log collection to complete.
2026-07-23T14:19:39.120386141Z Wrote inspect data to must-gather.
2026-07-23T14:19:39.125728629Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-23T14:19:39.235877571Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-23T14:19:39.404031894Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-23T14:19:39.423872564Z No resources found
2026-07-23T14:19:39.668128966Z ERROR: No running kube-apiserver pods found
2026-07-23T14:19:39.828285627Z Wrote inspect data to must-gather.
2026-07-23T14:19:39.967303474Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-23T14:19:40.002406127Z Wrote inspect data to must-gather.
2026-07-23T14:19:40.016227076Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-23T14:19:40.052810617Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-23T14:19:40.086345763Z Gathering data for ns/openshift-monitoring...
2026-07-23T14:19:40.152534546Z Wrote inspect data to must-gather.
2026-07-23T14:19:40.165763642Z INFO: 'previous-logs' folder not found on ip-10-0-138-123.ec2.internal, skipping...
2026-07-23T14:19:40.201127503Z Wrote inspect data to must-gather.
2026-07-23T14:19:40.206103775Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-23T14:19:40.243312424Z Wrote inspect data to must-gather.
2026-07-23T14:19:40.243753811Z tar: Removing leading `/' from member names
2026-07-23T14:19:40.249282574Z INFO: HAProxy config collection complete.
2026-07-23T14:19:40.302220133Z INFO: Getting rules from prometheus-k8s-0
2026-07-23T14:19:40.350630908Z error: the server doesn't have a resource type "machineconfigs"
2026-07-23T14:19:40.451615030Z INFO: 'previous-logs' folder not found on ip-10-0-140-244.ec2.internal, skipping...
2026-07-23T14:19:40.524007642Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-23T14:19:40.524402717Z Wrote inspect data to must-gather.
2026-07-23T14:19:40.743801901Z INFO: Getting status/config from prometheus-k8s-0
2026-07-23T14:19:40.750987691Z INFO: 'previous-logs' folder not found on ip-10-0-141-22.ec2.internal, skipping...
2026-07-23T14:19:40.751106009Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-23T14:19:40.751130563Z INFO: Machine Config Daemon termination log collection complete.
2026-07-23T14:19:40.778157865Z Wrote inspect data to must-gather.
2026-07-23T14:19:40.792243895Z Wrote inspect data to must-gather.
2026-07-23T14:19:40.939029637Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-23T14:19:41.036285001Z Wrote inspect data to must-gather.
2026-07-23T14:19:41.161826378Z Wrote inspect data to must-gather.
2026-07-23T14:19:41.192886796Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-23T14:19:41.319972437Z INFO: INTERCONNECT MODE
2026-07-23T14:19:41.320002317Z INFO: Gathering ovn-kubernetes DBs
2026-07-23T14:19:41.369995605Z Wrote inspect data to must-gather.
2026-07-23T14:19:41.439467559Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-23T14:19:41.457423028Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-23T14:19:41.482633499Z INFO: Gathering OVN_Northbound from ovnkube-node-cgj4x...
2026-07-23T14:19:41.482873485Z INFO: Gathering OVN_Northbound from ovnkube-node-dzj8t...
2026-07-23T14:19:41.483102698Z INFO: Gathering OVN_Northbound from ovnkube-node-rvskj...
2026-07-23T14:19:41.483392352Z INFO: Gathering OVN_Southbound from ovnkube-node-cgj4x...
2026-07-23T14:19:41.483682413Z INFO: Gathering OVN_Southbound from ovnkube-node-dzj8t...
2026-07-23T14:19:41.483989541Z INFO: Gathering OVN_Southbound from ovnkube-node-rvskj...
2026-07-23T14:19:41.620886591Z Wrote inspect data to must-gather.
2026-07-23T14:19:41.777835723Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-23T14:19:41.807957693Z tar: Removing leading `/' from member names
2026-07-23T14:19:41.826779851Z tar: Removing leading `/' from member names
2026-07-23T14:19:41.836264161Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:41.872622349Z tar: Removing leading `/' from member names
2026-07-23T14:19:41.872731653Z tar: Removing leading `/' from member names
2026-07-23T14:19:41.889931027Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-23T14:19:41.896780393Z tar: Removing leading `/' from member names
2026-07-23T14:19:41.909178764Z tar: Removing leading `/' from member names
2026-07-23T14:19:41.957667208Z Wrote inspect data to must-gather.
2026-07-23T14:19:42.072212892Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-23T14:19:42.113628945Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:42.116383012Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-23T14:19:42.216451306Z Wrote inspect data to must-gather.
2026-07-23T14:19:42.352730599Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:42.407179830Z INFO: Waiting for network log collection to complete ...
2026-07-23T14:19:42.407223077Z INFO: Waiting for ovnk database copies to complete ...
2026-07-23T14:19:42.407608272Z INFO: Copying ovnk databases complete.
2026-07-23T14:19:42.409251283Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-07-23T14:19:42.412340540Z ovnk_database_store/
2026-07-23T14:19:42.412389136Z ovnk_database_store/ovnkube-node-cgj4x_sbdb
2026-07-23T14:19:42.433381042Z ovnk_database_store/ovnkube-node-cgj4x_nbdb
2026-07-23T14:19:42.439145885Z ovnk_database_store/ovnkube-node-rvskj_nbdb
2026-07-23T14:19:42.445745905Z ovnk_database_store/ovnkube-node-rvskj_sbdb
2026-07-23T14:19:42.461034669Z W0723 14:19:42.460986    1592 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-23T14:19:42.463899053Z ovnk_database_store/ovnkube-node-dzj8t_nbdb
2026-07-23T14:19:42.467991488Z Wrote inspect data to must-gather.
2026-07-23T14:19:42.471818828Z ovnk_database_store/ovnkube-node-dzj8t_sbdb
2026-07-23T14:19:42.496203722Z INFO: Network log collection complete.
2026-07-23T14:19:42.526052192Z INFO: Getting status from alertmanager-main-0
2026-07-23T14:19:42.572207885Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-23T14:19:42.596573946Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:42.599830338Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-23T14:19:42.754684580Z Gathering data for ns/openshift-network-console...
2026-07-23T14:19:42.765476131Z Wrote inspect data to must-gather.
2026-07-23T14:19:42.827602883Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:43.033952728Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-23T14:19:43.037189428Z INFO: Fallback to identify the container image from release info
2026-07-23T14:19:43.040007041Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:43.043161366Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-23T14:19:43.196940064Z Gathering data for ns/openshift-console-operator...
2026-07-23T14:19:43.260555210Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:43.447801783Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:43.450115170Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-23T14:19:43.572097903Z Gathering data for ns/openshift-console...
2026-07-23T14:19:43.630422115Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:43.642791908Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-23T14:19:43.807119848Z daemonset.apps/perf-node-gather-daemonset created
2026-07-23T14:19:43.870544909Z Wrote inspect data to /must-gather/istio.
2026-07-23T14:19:43.936200011Z Waiting for performance profile collector pods to become ready: 1
2026-07-23T14:19:44.264913827Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-23T14:19:44.579657937Z Done executing Istio gather script
2026-07-23T14:19:44.767100297Z Gathering data for ns/openshift-dns-operator...
2026-07-23T14:19:45.021761283Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-23T14:19:45.036050715Z Gathering data for ns/openshift-dns...
2026-07-23T14:19:45.059265028Z Waiting for performance profile collector pods to become ready: 2
2026-07-23T14:19:45.586785762Z Gathering data for ns/openshift-image-registry...
2026-07-23T14:19:46.019553797Z Gathering data for ns/openshift-ingress-operator...
2026-07-23T14:19:46.169269261Z Waiting for performance profile collector pods to become ready: 3
2026-07-23T14:19:46.344995565Z Gathering data for ns/openshift-ingress...
2026-07-23T14:19:46.664689734Z Gathering data for ns/openshift-ingress-canary...
2026-07-23T14:19:47.028206319Z Gathering data for ns/openshift-insights...
2026-07-23T14:19:47.272234082Z Waiting for performance profile collector pods to become ready: 4
2026-07-23T14:19:48.383398652Z Waiting for performance profile collector pods to become ready: 5
2026-07-23T14:19:48.896354454Z Gathering data for ns/openshift-jobset-operator...
2026-07-23T14:19:49.492345878Z Waiting for performance profile collector pods to become ready: 6
2026-07-23T14:19:49.664343036Z Gathering data for ns/openshift-config...
2026-07-23T14:19:49.901659258Z Gathering data for ns/openshift-config-managed...
2026-07-23T14:19:50.028010089Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-23T14:19:50.306521769Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-23T14:19:50.563060932Z Gathering data for ns/openshift-kube-apiserver...
2026-07-23T14:19:50.620137957Z Waiting for performance profile collector pods to become ready: 7
2026-07-23T14:19:50.813256578Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-23T14:19:51.085664948Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-23T14:19:51.359648218Z Gathering data for ns/openshift-kube-scheduler...
2026-07-23T14:19:51.648269795Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-23T14:19:51.751594703Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-23T14:19:51.974550001Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-23T14:19:52.000031224Z Collecting performance related data for node ip-10-0-138-123.ec2.internal
2026-07-23T14:19:52.006555713Z Collecting performance related data for node ip-10-0-141-22.ec2.internal
2026-07-23T14:19:52.014553691Z Collecting performance related data for node ip-10-0-140-244.ec2.internal
2026-07-23T14:19:52.380745359Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-23T14:19:52.745187806Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-23T14:19:53.350436402Z Gathering data for ns/openshift-multus...
2026-07-23T14:19:54.731108634Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-23T14:19:55.034115825Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-23T14:19:55.965291292Z Gathering data for ns/openshift-host-network...
2026-07-23T14:19:56.279071470Z Gathering data for ns/openshift-network-diagnostics...
2026-07-23T14:19:56.826197140Z Gathering data for ns/openshift-network-node-identity...
2026-07-23T14:19:57.221641443Z Gathering data for ns/openshift-network-operator...
2026-07-23T14:19:57.604347851Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-23T14:19:57.866634927Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-23T14:19:58.322113606Z Gathering data for ns/openshift-apiserver-operator...
2026-07-23T14:19:58.605684570Z Gathering data for ns/openshift-apiserver...
2026-07-23T14:19:58.971392260Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-23T14:19:59.242730785Z Gathering data for ns/openshift-controller-manager...
2026-07-23T14:19:59.527287961Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-23T14:20:00.035406720Z Collecting kubelet logs for node ip-10-0-138-123.ec2.internal
2026-07-23T14:20:00.037605465Z Collecting kubelet logs for node ip-10-0-141-22.ec2.internal
2026-07-23T14:20:00.039348048Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-23T14:20:00.039457724Z Collecting kubelet logs for node ip-10-0-140-244.ec2.internal
2026-07-23T14:20:00.205824479Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-23T14:20:00.363271234Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-nvwjt namespace
2026-07-23T14:20:00.369052735Z INFO: Node performance data collection complete.
2026-07-23T14:20:00.503486716Z Gathering data for ns/openshift-service-ca-operator...
2026-07-23T14:20:00.808554201Z Gathering data for ns/openshift-service-ca...
2026-07-23T14:20:01.154566979Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-23T14:20:02.004258971Z Wrote inspect data to must-gather.
2026-07-23T14:20:02.004307578Z error: inspection completed with the errors occurred while gathering data:
2026-07-23T14:20:02.004307578Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-23T14:20:02.126638283Z error: the server doesn't have a resource type "clusters"
2026-07-23T14:20:02.249368630Z Caches written to disk