2026-07-06T19:31:47.470640120Z [disk usage checker] Started
2026-07-06T19:31:47.473908073Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T19:31:47.672861019Z Gathering data for ns/openshift-cluster-version...
2026-07-06T19:31:47.816787768Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-06T19:31:47.918098389Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T19:31:48.005414132Z Gathering data for ns/default...
2026-07-06T19:31:48.328500753Z Gathering data for ns/openshift...
2026-07-06T19:31:48.753636963Z Gathering data for ns/kube-system...
2026-07-06T19:31:49.513722215Z Gathering data for ns/openshift-etcd...
2026-07-06T19:31:49.879816164Z Wrote inspect data to must-gather.
2026-07-06T19:31:49.879850401Z error: inspection completed with the errors occurred while gathering data:
2026-07-06T19:31:49.879850401Z     namespaces "assisted-installer" not found
2026-07-06T19:31:50.281969806Z Waiting on subprocesses to finish execution.
2026-07-06T19:31:50.284519497Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-06T19:31:50.294999982Z Executing Istio gather script
2026-07-06T19:31:50.302376094Z INFO: Collecting host service logs for crio
2026-07-06T19:31:50.302713257Z INFO: Collecting host service logs for kubelet
2026-07-06T19:31:50.303029654Z INFO: Collecting host service logs for rpm-ostreed
2026-07-06T19:31:50.303319991Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-06T19:31:50.303605014Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-06T19:31:50.303906707Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-06T19:31:50.304198006Z INFO: Collecting host service logs for NetworkManager
2026-07-06T19:31:50.304483925Z INFO: Collecting host service logs for openvswitch
2026-07-06T19:31:50.304770088Z INFO: Collecting host service logs for ovs-configuration
2026-07-06T19:31:50.305077563Z INFO: Collecting host service logs for ovsdb-server
2026-07-06T19:31:50.305380201Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-06T19:31:50.305717257Z INFO: Waiting for worker host service log collection to complete ...
2026-07-06T19:31:50.315947814Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-06T19:31:50.323919886Z INFO: Gathering HAProxy config files
2026-07-06T19:31:50.334161301Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-06T19:31:50.459029302Z INFO: Waiting for node performance related collection to complete ...
2026-07-06T19:31:51.128853218Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-06T19:31:51.128956579Z INFO: on-disk MachineConfig config collection complete.
2026-07-06T19:31:51.219856002Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-06T19:31:51.223510007Z No resources found
2026-07-06T19:31:51.226618215Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-06T19:31:51.308927356Z No resources found in openshift-etcd namespace.
2026-07-06T19:31:51.354065484Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-06T19:31:51.436170486Z INFO: "metallb-operator" not detected. Skipping.
2026-07-06T19:31:51.664043008Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-xh8cv
2026-07-06T19:31:51.664090544Z insights-runtime-extractor-6smmm
2026-07-06T19:31:51.664124264Z insights-runtime-extractor-l8lrj
2026-07-06T19:31:51.664144859Z insights-runtime-extractor-p4vsp
2026-07-06T19:31:51.667590405Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-06T19:31:51.742340416Z error: the server doesn't have a resource type "performanceprofile"
2026-07-06T19:31:51.974858126Z Wrote inspect data to must-gather.
2026-07-06T19:31:51.991339052Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-06T19:31:52.031113072Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-06T19:31:52.031113072Z See 'oc rsync -h' for help and examples
2026-07-06T19:31:52.111969665Z INFO: Worker host service log collection to complete.
2026-07-06T19:31:52.145770705Z Wrote inspect data to must-gather.
2026-07-06T19:31:52.339962279Z ERROR: No running kube-apiserver pods found
2026-07-06T19:31:52.473224783Z No resources found
2026-07-06T19:31:52.525563978Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-06T19:31:52.580653109Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-06T19:31:52.694670099Z Gathering data for ns/openshift-monitoring...
2026-07-06T19:31:52.803033530Z INFO: 'previous-logs' folder not found on ip-10-0-133-192.ec2.internal, skipping...
2026-07-06T19:31:52.899583076Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-06T19:31:52.914670972Z Wrote inspect data to must-gather.
2026-07-06T19:31:52.982306540Z Wrote inspect data to must-gather.
2026-07-06T19:31:53.056227076Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T19:31:53.163194245Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-06T19:31:53.169391460Z Wrote inspect data to must-gather.
2026-07-06T19:31:53.170239426Z Wrote inspect data to must-gather.
2026-07-06T19:31:53.202601005Z Wrote inspect data to must-gather.
2026-07-06T19:31:53.244233051Z INFO: Getting rules from prometheus-k8s-0
2026-07-06T19:31:53.299813166Z tar: Removing leading `/' from member names
2026-07-06T19:31:53.308247851Z INFO: HAProxy config collection complete.
2026-07-06T19:31:53.339321118Z error: the server doesn't have a resource type "machineconfigs"
2026-07-06T19:31:53.386589690Z INFO: 'previous-logs' folder not found on ip-10-0-135-162.ec2.internal, skipping...
2026-07-06T19:31:53.513617814Z Wrote inspect data to must-gather.
2026-07-06T19:31:53.558621840Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-06T19:31:53.712062586Z INFO: 'previous-logs' folder not found on ip-10-0-136-186.ec2.internal, skipping...
2026-07-06T19:31:53.712157896Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-06T19:31:53.712213636Z INFO: Machine Config Daemon termination log collection complete.
2026-07-06T19:31:53.715031241Z INFO: Getting status/config from prometheus-k8s-0
2026-07-06T19:31:53.829971459Z Wrote inspect data to must-gather.
2026-07-06T19:31:53.843554472Z Wrote inspect data to must-gather.
2026-07-06T19:31:53.994789563Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-06T19:31:54.105548925Z Wrote inspect data to must-gather.
2026-07-06T19:31:54.146232461Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-06T19:31:54.182131602Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-06T19:31:54.272852850Z Wrote inspect data to must-gather.
2026-07-06T19:31:54.417210691Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:54.421089097Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-06T19:31:54.431188477Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-06T19:31:54.472380850Z INFO: INTERCONNECT MODE
2026-07-06T19:31:54.472467449Z INFO: Gathering ovn-kubernetes DBs
2026-07-06T19:31:54.554952342Z Wrote inspect data to must-gather.
2026-07-06T19:31:54.685358984Z INFO: Gathering OVN_Northbound from ovnkube-node-6f6ss...
2026-07-06T19:31:54.685640585Z INFO: Gathering OVN_Northbound from ovnkube-node-8qg9j...
2026-07-06T19:31:54.685890664Z INFO: Gathering OVN_Northbound from ovnkube-node-r2gxh...
2026-07-06T19:31:54.686160621Z INFO: Gathering OVN_Southbound from ovnkube-node-6f6ss...
2026-07-06T19:31:54.686402991Z INFO: Gathering OVN_Southbound from ovnkube-node-8qg9j...
2026-07-06T19:31:54.687020799Z INFO: Gathering OVN_Southbound from ovnkube-node-r2gxh...
2026-07-06T19:31:54.709703792Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:54.713221937Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-06T19:31:54.719649437Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-06T19:31:54.852890462Z Wrote inspect data to must-gather.
2026-07-06T19:31:55.009207181Z tar: Removing leading `/' from member names
2026-07-06T19:31:55.087162635Z tar: Removing leading `/' from member names
2026-07-06T19:31:55.093585779Z tar: Removing leading `/' from member names
2026-07-06T19:31:55.098188279Z tar: Removing leading `/' from member names
2026-07-06T19:31:55.114848470Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:55.129346451Z tar: Removing leading `/' from member names
2026-07-06T19:31:55.161163038Z tar: Removing leading `/' from member names
2026-07-06T19:31:55.168840908Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-06T19:31:55.243623975Z W0706 19:31:55.243573    1405 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-06T19:31:55.251150484Z Wrote inspect data to must-gather.
2026-07-06T19:31:55.362780196Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T19:31:55.402387741Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:55.405781102Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-06T19:31:55.621038669Z INFO: Getting status from alertmanager-main-0
2026-07-06T19:31:55.643636830Z Gathering data for ns/openshift-network-console...
2026-07-06T19:31:55.651728737Z Wrote inspect data to must-gather.
2026-07-06T19:31:55.692713055Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:55.742045384Z INFO: Waiting for network log collection to complete ...
2026-07-06T19:31:55.742108543Z INFO: Waiting for ovnk database copies to complete ...
2026-07-06T19:31:55.742641177Z INFO: Copying ovnk databases complete.
2026-07-06T19:31:55.746976239Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-07-06T19:31:55.751974817Z ovnk_database_store/
2026-07-06T19:31:55.752027062Z ovnk_database_store/ovnkube-node-6f6ss_nbdb
2026-07-06T19:31:55.764530124Z ovnk_database_store/ovnkube-node-6f6ss_sbdb
2026-07-06T19:31:55.786272904Z ovnk_database_store/ovnkube-node-r2gxh_sbdb
2026-07-06T19:31:55.806713174Z ovnk_database_store/ovnkube-node-r2gxh_nbdb
2026-07-06T19:31:55.815332488Z ovnk_database_store/ovnkube-node-8qg9j_nbdb
2026-07-06T19:31:55.825964057Z ovnk_database_store/ovnkube-node-8qg9j_sbdb
2026-07-06T19:31:55.852307692Z INFO: Network log collection complete.
2026-07-06T19:31:55.952548182Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:55.955251214Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-06T19:31:55.988669594Z Wrote inspect data to must-gather.
2026-07-06T19:31:56.207285324Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:56.311545312Z Gathering data for ns/openshift-console-operator...
2026-07-06T19:31:56.325618460Z Wrote inspect data to must-gather.
2026-07-06T19:31:56.445445529Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:56.448809213Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-06T19:31:56.602488787Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-06T19:31:56.604736970Z INFO: Fallback to identify the container image from release info
2026-07-06T19:31:56.648871849Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:56.866810013Z Gathering data for ns/openshift-console...
2026-07-06T19:31:56.877615489Z Wrote inspect data to /must-gather/istio.
2026-07-06T19:31:57.358266913Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-06T19:31:57.483581884Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-06T19:31:57.493306198Z daemonset.apps/perf-node-gather-daemonset created
2026-07-06T19:31:57.598080102Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-06T19:31:57.598144121Z Done executing Istio gather script
2026-07-06T19:31:57.629470986Z Waiting for performance profile collector pods to become ready: 1
2026-07-06T19:31:58.001303611Z Gathering data for ns/openshift-dns-operator...
2026-07-06T19:31:58.339976134Z Gathering data for ns/openshift-dns...
2026-07-06T19:31:58.738491361Z Waiting for performance profile collector pods to become ready: 2
2026-07-06T19:31:58.927939033Z Gathering data for ns/openshift-image-registry...
2026-07-06T19:31:59.380887813Z Gathering data for ns/openshift-ingress-operator...
2026-07-06T19:31:59.746863172Z Gathering data for ns/openshift-ingress...
2026-07-06T19:31:59.860586796Z Waiting for performance profile collector pods to become ready: 3
2026-07-06T19:32:00.084832905Z Gathering data for ns/openshift-ingress-canary...
2026-07-06T19:32:00.494753790Z Gathering data for ns/openshift-insights...
2026-07-06T19:32:00.982729405Z Waiting for performance profile collector pods to become ready: 4
2026-07-06T19:32:02.094286058Z Waiting for performance profile collector pods to become ready: 5
2026-07-06T19:32:02.453048641Z Gathering data for ns/openshift-config...
2026-07-06T19:32:02.604492535Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-06T19:32:02.707697172Z Gathering data for ns/openshift-config-managed...
2026-07-06T19:32:03.136077958Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-06T19:32:03.223786315Z Waiting for performance profile collector pods to become ready: 6
2026-07-06T19:32:03.436470838Z Gathering data for ns/openshift-kube-apiserver...
2026-07-06T19:32:03.735457209Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-06T19:32:04.043602377Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-06T19:32:04.353606023Z Waiting for performance profile collector pods to become ready: 7
2026-07-06T19:32:04.368268644Z Gathering data for ns/openshift-kube-scheduler...
2026-07-06T19:32:04.683031327Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-06T19:32:05.020817973Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-06T19:32:05.442600027Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-06T19:32:05.489814048Z Waiting for performance profile collector pods to become ready: 8
2026-07-06T19:32:05.821206663Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-06T19:32:06.372563008Z Gathering data for ns/openshift-multus...
2026-07-06T19:32:06.595023087Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-06T19:32:06.835168085Z Collecting performance related data for node ip-10-0-133-192.ec2.internal
2026-07-06T19:32:06.842935525Z Collecting performance related data for node ip-10-0-135-162.ec2.internal
2026-07-06T19:32:06.850975295Z Collecting performance related data for node ip-10-0-136-186.ec2.internal
2026-07-06T19:32:07.613454490Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-06T19:32:08.039266668Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-06T19:32:09.444007927Z Gathering data for ns/openshift-host-network...
2026-07-06T19:32:09.884279379Z Gathering data for ns/openshift-network-diagnostics...
2026-07-06T19:32:10.519332183Z Gathering data for ns/openshift-network-node-identity...
2026-07-06T19:32:10.986501934Z Gathering data for ns/openshift-network-operator...
2026-07-06T19:32:11.413485030Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-06T19:32:11.718204595Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-06T19:32:12.271850364Z Gathering data for ns/openshift-apiserver-operator...
2026-07-06T19:32:12.615000701Z Gathering data for ns/openshift-apiserver...
2026-07-06T19:32:12.618672838Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-06T19:32:13.080103949Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-06T19:32:13.400207313Z Gathering data for ns/openshift-controller-manager...
2026-07-06T19:32:13.685202208Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-06T19:32:14.399870408Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-06T19:32:14.706169322Z Gathering data for ns/openshift-service-ca-operator...
2026-07-06T19:32:15.069361226Z Gathering data for ns/openshift-service-ca...
2026-07-06T19:32:15.515078903Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-06T19:32:15.656673714Z Collecting kubelet logs for node ip-10-0-133-192.ec2.internal
2026-07-06T19:32:15.658396199Z Collecting kubelet logs for node ip-10-0-135-162.ec2.internal
2026-07-06T19:32:15.660159760Z Collecting kubelet logs for node ip-10-0-136-186.ec2.internal
2026-07-06T19:32:16.004802285Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-wgzzb namespace
2026-07-06T19:32:16.014986766Z INFO: Node performance data collection complete.
2026-07-06T19:32:16.492157072Z Wrote inspect data to must-gather.
2026-07-06T19:32:16.492198179Z error: inspection completed with the errors occurred while gathering data:
2026-07-06T19:32:16.492198179Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-06T19:32:16.610760223Z error: the server doesn't have a resource type "clusters"
2026-07-06T19:32:17.623345931Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-06T19:32:21.254357801Z Caches written to disk