2026-10-30T11:36:33.593212300Z [disk usage checker] Started
2026-10-30T11:36:33.596588204Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:36:33.983326050Z Gathering data for ns/openshift-cluster-version...
2026-10-30T11:36:34.308280615Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-30T11:36:34.514793173Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-30T11:36:34.709553892Z Gathering data for ns/default...
2026-10-30T11:36:35.421725190Z Gathering data for ns/openshift...
2026-10-30T11:36:36.218558337Z Gathering data for ns/kube-system...
2026-10-30T11:36:37.119766232Z Waiting on subprocesses to finish execution.
2026-10-30T11:36:37.128274222Z INFO: Gathering machine config daemon's old logs from all nodes
2026-10-30T11:36:37.133436514Z INFO: Gathering HAProxy config files
2026-10-30T11:36:37.144332300Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-10-30T11:36:37.149632161Z Executing Istio gather script
2026-10-30T11:36:37.163028278Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-10-30T11:36:37.183186198Z INFO: Collecting host service logs for crio
2026-10-30T11:36:37.183644062Z INFO: Collecting host service logs for kubelet
2026-10-30T11:36:37.183984577Z INFO: Collecting host service logs for rpm-ostreed
2026-10-30T11:36:37.184301562Z INFO: Collecting host service logs for ostree-finalize-staged
2026-10-30T11:36:37.184651151Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-10-30T11:36:37.184965638Z INFO: Collecting host service logs for machine-config-daemon-host
2026-10-30T11:36:37.185276361Z INFO: Collecting host service logs for NetworkManager
2026-10-30T11:36:37.185606732Z INFO: Collecting host service logs for openvswitch
2026-10-30T11:36:37.185926908Z INFO: Collecting host service logs for ovs-configuration
2026-10-30T11:36:37.186249604Z INFO: Collecting host service logs for ovsdb-server
2026-10-30T11:36:37.186593884Z INFO: Collecting host service logs for ovs-vswitchd
2026-10-30T11:36:37.186991715Z INFO: Waiting for worker host service log collection to complete ...
2026-10-30T11:36:37.250203263Z INFO: Waiting for node performance related collection to complete ...
2026-10-30T11:36:37.828131416Z INFO: "metallb-operator" not detected. Skipping.
2026-10-30T11:36:37.888461027Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-10-30T11:36:37.888542410Z INFO: on-disk MachineConfig config collection complete.
2026-10-30T11:36:37.898189688Z Wrote inspect data to must-gather.
2026-10-30T11:36:37.915770215Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-10-30T11:36:37.943669517Z INFO: "nfd" not detected. Skipping.
2026-10-30T11:36:37.988233829Z INFO: Found 1 replicas - prometheus-k8s-0
2026-10-30T11:36:38.159781667Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-10-30T11:36:38.242186334Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-v69fn
2026-10-30T11:36:38.242186334Z insights-runtime-extractor-2kxlm
2026-10-30T11:36:38.242186334Z insights-runtime-extractor-cn9sm
2026-10-30T11:36:38.242186334Z insights-runtime-extractor-dpvzb
2026-10-30T11:36:38.397560603Z ERROR: No running kube-apiserver pods found
2026-10-30T11:36:38.595314967Z No resources found
2026-10-30T11:36:38.702092216Z No resources found in openshift-etcd namespace.
2026-10-30T11:36:38.747248617Z error: the server doesn't have a resource type "performanceprofile"
2026-10-30T11:36:38.755217777Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-10-30T11:36:38.755217777Z See 'oc rsync -h' for help and examples
2026-10-30T11:36:38.772167170Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:36:38.843075586Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-10-30T11:36:38.927149561Z INFO: "sriov-network-operator" not detected. Skipping.
2026-10-30T11:36:38.982632934Z INFO: Worker host service log collection to complete.
2026-10-30T11:36:39.188843134Z INFO: Waiting for HAProxy config collection to complete ...
2026-10-30T11:36:39.597285780Z INFO: 'previous-logs' folder not found on ip-10-0-132-202.ec2.internal, skipping...
2026-10-30T11:36:39.640536156Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-30T11:36:39.689849250Z Wrote inspect data to must-gather.
2026-10-30T11:36:39.734374169Z Wrote inspect data to must-gather.
2026-10-30T11:36:39.908458895Z Wrote inspect data to must-gather.
2026-10-30T11:36:39.942246804Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-10-30T11:36:39.961111704Z tar: Removing leading `/' from member names
2026-10-30T11:36:39.990287010Z INFO: HAProxy config collection complete.
2026-10-30T11:36:40.046546183Z W1030 11:36:40.046498     768 util.go:224] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-10-30T11:36:40.058265266Z Wrote inspect data to must-gather.
2026-10-30T11:36:40.111129219Z No resources found
2026-10-30T11:36:40.162324913Z Wrote inspect data to must-gather.
2026-10-30T11:36:40.468558823Z error: the server doesn't have a resource type "machineconfigs"
2026-10-30T11:36:40.528489343Z INFO: Getting rules from prometheus-k8s-0
2026-10-30T11:36:40.555790809Z INFO: 'previous-logs' folder not found on ip-10-0-137-242.ec2.internal, skipping...
2026-10-30T11:36:40.634115834Z Gathering data for ns/openshift-etcd...
2026-10-30T11:36:40.798797558Z Wrote inspect data to must-gather.
2026-10-30T11:36:40.865313007Z error: the server doesn't have a resource type "machineconfigpools"
2026-10-30T11:36:40.905829768Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-10-30T11:36:41.047370801Z Gathering data for ns/openshift-cloud-credential-operator...
2026-10-30T11:36:41.104695676Z INFO: 'previous-logs' folder not found on ip-10-0-140-230.ec2.internal, skipping...
2026-10-30T11:36:41.104729808Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-10-30T11:36:41.104759293Z INFO: Machine Config Daemon termination log collection complete.
2026-10-30T11:36:41.337980407Z Wrote inspect data to must-gather.
2026-10-30T11:36:41.339749067Z Wrote inspect data to must-gather.
2026-10-30T11:36:41.440568002Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-30T11:36:41.510213267Z INFO: Getting status/config from prometheus-k8s-0
2026-10-30T11:36:41.651848121Z error: the server doesn't have a resource type "kubeletconfigs"
2026-10-30T11:36:41.785038675Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-30T11:36:41.851579044Z Wrote inspect data to must-gather.
2026-10-30T11:36:41.851676078Z error: inspection completed with the errors occurred while gathering data:
2026-10-30T11:36:41.851676078Z     namespaces "assisted-installer" not found
2026-10-30T11:36:41.896713042Z Wrote inspect data to must-gather.
2026-10-30T11:36:42.084707902Z Wrote inspect data to must-gather.
2026-10-30T11:36:42.309593172Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-10-30T11:36:42.359574283Z INFO: Getting status/flags from prometheus-k8s-0
2026-10-30T11:36:42.407091479Z INFO: LEGACY MODE
2026-10-30T11:36:42.407132852Z INFO: Gathering ovn-kubernetes master data
2026-10-30T11:36:42.481423420Z Wrote inspect data to must-gather.
2026-10-30T11:36:42.686965772Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:42.752948990Z Gathering data for ns/openshift-monitoring...
2026-10-30T11:36:42.813270484Z Wrote inspect data to must-gather.
2026-10-30T11:36:42.839196963Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-10-30T11:36:43.114413795Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:43.118070334Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-10-30T11:36:43.167398948Z Wrote inspect data to must-gather.
2026-10-30T11:36:43.276089403Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-10-30T11:36:43.444637327Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:43.521531172Z Wrote inspect data to must-gather.
2026-10-30T11:36:43.746882908Z INFO: Waiting for network log collection to complete ...
2026-10-30T11:36:43.746929094Z INFO: Waiting for ovnk database copies to complete ...
2026-10-30T11:36:43.747368374Z INFO: Copying ovnk databases complete.
2026-10-30T11:36:43.748992211Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 759 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 760 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 761 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 762 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 763 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 764 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 765 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 766 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 818 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 819 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 820 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 821 is not a child of this shell
2026-10-30T11:36:43.749623068Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 822 is not a child of this shell
2026-10-30T11:36:43.753433799Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-10-30T11:36:43.756814120Z tar: Exiting with failure status due to previous errors
2026-10-30T11:36:43.757474032Z INFO: Network log collection complete.
2026-10-30T11:36:43.797069051Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:36:43.800334744Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:43.804549327Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-10-30T11:36:43.819139000Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-10-30T11:36:43.880775324Z Wrote inspect data to must-gather.
2026-10-30T11:36:44.120453143Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:44.178528132Z Wrote inspect data to must-gather.
2026-10-30T11:36:44.412003180Z INFO: Getting status from alertmanager-main-0
2026-10-30T11:36:44.517826610Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:44.521818385Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-10-30T11:36:44.644555654Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-10-30T11:36:44.829763557Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:44.854622112Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-10-30T11:36:45.058813058Z daemonset.apps/perf-node-gather-daemonset created
2026-10-30T11:36:45.172904059Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:45.175965791Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-10-30T11:36:45.222703134Z Waiting for performance profile collector pods to become ready: 1
2026-10-30T11:36:45.484168508Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:45.826832569Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:45.829389062Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-10-30T11:36:46.049002135Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:46.360937292Z Wrote inspect data to /must-gather/istio.
2026-10-30T11:36:46.396288274Z Waiting for performance profile collector pods to become ready: 2
2026-10-30T11:36:46.434354370Z Wrote inspect data to must-gather.
2026-10-30T11:36:47.298847078Z Done executing Istio gather script
2026-10-30T11:36:47.536617457Z Waiting for performance profile collector pods to become ready: 3
2026-10-30T11:36:48.678848396Z Waiting for performance profile collector pods to become ready: 4
2026-10-30T11:36:48.802170003Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:36:49.820395040Z Waiting for performance profile collector pods to become ready: 5
2026-10-30T11:36:49.826969424Z Gathering data for ns/openshift-network-console...
2026-10-30T11:36:50.831185000Z Gathering data for ns/openshift-console-operator...
2026-10-30T11:36:50.961214734Z Waiting for performance profile collector pods to become ready: 6
2026-10-30T11:36:51.769160011Z Gathering data for ns/openshift-console...
2026-10-30T11:36:52.105502243Z Waiting for performance profile collector pods to become ready: 7
2026-10-30T11:36:52.971126067Z Gathering data for ns/openshift-cluster-storage-operator...
2026-10-30T11:36:53.263146085Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-10-30T11:36:53.563703911Z Collecting performance related data for node ip-10-0-137-242.ec2.internal
2026-10-30T11:36:53.573293999Z Collecting performance related data for node ip-10-0-140-230.ec2.internal
2026-10-30T11:36:53.580244038Z Collecting performance related data for node ip-10-0-132-202.ec2.internal
2026-10-30T11:36:53.806523683Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:36:54.252550025Z Gathering data for ns/openshift-dns-operator...
2026-10-30T11:36:55.113423095Z Gathering data for ns/openshift-dns...
2026-10-30T11:36:57.294287039Z Gathering data for ns/openshift-image-registry...
2026-10-30T11:36:58.655941303Z Gathering data for ns/openshift-ingress-operator...
2026-10-30T11:36:58.810965851Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:36:59.458980334Z Gathering data for ns/openshift-ingress...
2026-10-30T11:37:00.361059627Z Gathering data for ns/openshift-ingress-canary...
2026-10-30T11:37:01.561017665Z Gathering data for ns/openshift-insights...
2026-10-30T11:37:01.710634625Z Collecting kubelet logs for node ip-10-0-137-242.ec2.internal
2026-10-30T11:37:01.712520390Z Collecting kubelet logs for node ip-10-0-140-230.ec2.internal
2026-10-30T11:37:01.714178322Z Collecting kubelet logs for node ip-10-0-132-202.ec2.internal
2026-10-30T11:37:02.318220059Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-tgn5g namespace
2026-10-30T11:37:02.338109791Z INFO: Node performance data collection complete.
2026-10-30T11:37:03.815360212Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:04.834701316Z Gathering data for ns/openshift-operators-redhat...
2026-10-30T11:37:06.598805919Z Gathering data for ns/openshift-opentelemetry-operator...
2026-10-30T11:37:07.877259884Z Gathering data for ns/openshift-cluster-observability-operator...
2026-10-30T11:37:08.819977987Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:10.017001126Z Gathering data for ns/openshift-config...
2026-10-30T11:37:10.734367231Z Gathering data for ns/openshift-config-managed...
2026-10-30T11:37:11.609280473Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-10-30T11:37:12.315137693Z Gathering data for ns/openshift-kube-apiserver...
2026-10-30T11:37:13.042213466Z Gathering data for ns/openshift-kube-controller-manager...
2026-10-30T11:37:13.797421007Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-10-30T11:37:13.824484004Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:14.592205813Z Gathering data for ns/openshift-kube-scheduler...
2026-10-30T11:37:15.323599979Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-10-30T11:37:16.112883969Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-10-30T11:37:17.120592938Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-10-30T11:37:17.985803051Z Gathering data for ns/openshift-user-workload-monitoring...
2026-10-30T11:37:18.828914297Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:19.129475030Z Gathering data for ns/openshift-multus...
2026-10-30T11:37:23.833449853Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:25.145797106Z Gathering data for ns/openshift-ovn-kubernetes...
2026-10-30T11:37:28.837867767Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:30.145794168Z Gathering data for ns/openshift-host-network...
2026-10-30T11:37:30.985425237Z Gathering data for ns/openshift-network-diagnostics...
2026-10-30T11:37:32.701013242Z Gathering data for ns/openshift-network-node-identity...
2026-10-30T11:37:33.843455274Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:33.924276910Z Gathering data for ns/openshift-network-operator...
2026-10-30T11:37:35.084832658Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-10-30T11:37:35.800196907Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-10-30T11:37:37.171250761Z Gathering data for ns/openshift-apiserver-operator...
2026-10-30T11:37:37.892275993Z Gathering data for ns/openshift-apiserver...
2026-10-30T11:37:38.847872064Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:38.901219252Z Gathering data for ns/openshift-controller-manager-operator...
2026-10-30T11:37:39.603277313Z Gathering data for ns/openshift-controller-manager...
2026-10-30T11:37:40.337526745Z Gathering data for ns/openshift-cluster-samples-operator...
2026-10-30T11:37:41.680746956Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-10-30T11:37:42.440045714Z Gathering data for ns/openshift-service-ca-operator...
2026-10-30T11:37:43.310026204Z Gathering data for ns/openshift-service-ca...
2026-10-30T11:37:43.853362768Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-10-30T11:37:44.303337678Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-10-30T11:37:46.642886999Z Wrote inspect data to must-gather.
2026-10-30T11:37:46.642928017Z error: inspection completed with the errors occurred while gathering data:
2026-10-30T11:37:46.642928017Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-10-30T11:37:46.849264090Z error: the server doesn't have a resource type "clusters"
2026-10-30T11:37:46.927734833Z Caches written to disk