2026-10-31T01:10:30.060010748Z [disk usage checker] Started
2026-10-31T01:10:30.063377547Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-10-31T01:10:30.275590654Z Gathering data for ns/openshift-cluster-version...
2026-10-31T01:10:30.430479505Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-31T01:10:30.530820629Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-31T01:10:30.639997072Z Gathering data for ns/default...
2026-10-31T01:10:30.963461925Z Gathering data for ns/openshift...
2026-10-31T01:10:31.406504544Z Gathering data for ns/kube-system...
2026-10-31T01:10:32.376142045Z Gathering data for ns/openshift-etcd...
2026-10-31T01:10:32.709211535Z Waiting on subprocesses to finish execution.
2026-10-31T01:10:32.714150451Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-10-31T01:10:32.722789147Z INFO: Gathering machine config daemon's old logs from all nodes
2026-10-31T01:10:32.723522058Z INFO: Gathering HAProxy config files
2026-10-31T01:10:32.729046652Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-10-31T01:10:32.737583011Z Executing Istio gather script
2026-10-31T01:10:32.755972172Z INFO: Waiting for node performance related collection to complete ...
2026-10-31T01:10:32.825821840Z Wrote inspect data to must-gather.
2026-10-31T01:10:32.825948351Z error: inspection completed with the errors occurred while gathering data:
2026-10-31T01:10:32.825948351Z     namespaces "assisted-installer" not found
2026-10-31T01:10:32.844239083Z INFO: Collecting host service logs for crio
2026-10-31T01:10:32.844501438Z INFO: Collecting host service logs for kubelet
2026-10-31T01:10:32.844741249Z INFO: Collecting host service logs for rpm-ostreed
2026-10-31T01:10:32.844978078Z INFO: Collecting host service logs for ostree-finalize-staged
2026-10-31T01:10:32.845221598Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-10-31T01:10:32.845439235Z INFO: Collecting host service logs for machine-config-daemon-host
2026-10-31T01:10:32.845665989Z INFO: Collecting host service logs for NetworkManager
2026-10-31T01:10:32.845896323Z INFO: Collecting host service logs for openvswitch
2026-10-31T01:10:32.846110710Z INFO: Collecting host service logs for ovs-configuration
2026-10-31T01:10:32.846350724Z INFO: Collecting host service logs for ovsdb-server
2026-10-31T01:10:32.846563145Z INFO: Collecting host service logs for ovs-vswitchd
2026-10-31T01:10:32.846832990Z INFO: Waiting for worker host service log collection to complete ...
2026-10-31T01:10:33.110261313Z No resources found
2026-10-31T01:10:33.322493393Z error: the server doesn't have a resource type "performanceprofile"
2026-10-31T01:10:33.429651063Z No resources found in openshift-etcd namespace.
2026-10-31T01:10:33.716416193Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-58wsl
2026-10-31T01:10:33.716416193Z insights-runtime-extractor-rf4ng
2026-10-31T01:10:33.716416193Z insights-runtime-extractor-sx67l
2026-10-31T01:10:33.716416193Z insights-runtime-extractor-wm8mc
2026-10-31T01:10:33.717764517Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-10-31T01:10:33.807128267Z INFO: "nfd" not detected. Skipping.
2026-10-31T01:10:33.960279198Z INFO: "sriov-network-operator" not detected. Skipping.
2026-10-31T01:10:34.022368639Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-10-31T01:10:34.022565765Z INFO: on-disk MachineConfig config collection complete.
2026-10-31T01:10:34.077188169Z INFO: Waiting for HAProxy config collection to complete ...
2026-10-31T01:10:34.244335083Z INFO: Worker host service log collection to complete.
2026-10-31T01:10:34.273099302Z INFO: Found 1 replicas - prometheus-k8s-0
2026-10-31T01:10:34.300631139Z Wrote inspect data to must-gather.
2026-10-31T01:10:34.426384590Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-10-31T01:10:34.443333533Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-10-31T01:10:34.443333533Z See 'oc rsync -h' for help and examples
2026-10-31T01:10:34.460911911Z INFO: "metallb-operator" not detected. Skipping.
2026-10-31T01:10:34.616953396Z tar: Removing leading `/' from member names
2026-10-31T01:10:34.622483616Z INFO: HAProxy config collection complete.
2026-10-31T01:10:34.632495890Z Wrote inspect data to must-gather.
2026-10-31T01:10:34.667386392Z error: the server doesn't have a resource type "machineconfigs"
2026-10-31T01:10:34.678205071Z ERROR: No running kube-apiserver pods found
2026-10-31T01:10:34.753043629Z Wrote inspect data to must-gather.
2026-10-31T01:10:34.967472291Z Gathering data for ns/openshift-cloud-credential-operator...
2026-10-31T01:10:34.978280056Z INFO: 'previous-logs' folder not found on ip-10-0-129-75.us-east-2.compute.internal, skipping...
2026-10-31T01:10:35.055718825Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-10-31T01:10:35.087014522Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-10-31T01:10:35.172504871Z Wrote inspect data to must-gather.
2026-10-31T01:10:35.250628834Z Wrote inspect data to must-gather.
2026-10-31T01:10:35.265761885Z No resources found
2026-10-31T01:10:35.349788315Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-10-31T01:10:35.378183009Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-10-31T01:10:35.523453448Z Wrote inspect data to must-gather.
2026-10-31T01:10:35.525803390Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-31T01:10:35.538239071Z Wrote inspect data to must-gather.
2026-10-31T01:10:35.583369787Z error: the server doesn't have a resource type "machineconfigpools"
2026-10-31T01:10:35.602155261Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-10-31T01:10:35.738800698Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-10-31T01:10:35.853377753Z INFO: 'previous-logs' folder not found on ip-10-0-135-10.us-east-2.compute.internal, skipping...
2026-10-31T01:10:35.892763898Z Wrote inspect data to must-gather.
2026-10-31T01:10:35.906975274Z INFO: Getting rules from prometheus-k8s-0
2026-10-31T01:10:35.913782908Z Wrote inspect data to must-gather.
2026-10-31T01:10:36.105626198Z error: the server doesn't have a resource type "kubeletconfigs"
2026-10-31T01:10:36.164688397Z Gathering data for ns/openshift-monitoring...
2026-10-31T01:10:36.196488108Z INFO: 'previous-logs' folder not found on ip-10-0-143-150.us-east-2.compute.internal, skipping...
2026-10-31T01:10:36.196521494Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-10-31T01:10:36.196554645Z INFO: Machine Config Daemon termination log collection complete.
2026-10-31T01:10:36.223682893Z Wrote inspect data to must-gather.
2026-10-31T01:10:36.344741863Z INFO: Getting status/config from prometheus-k8s-0
2026-10-31T01:10:36.386537013Z Wrote inspect data to must-gather.
2026-10-31T01:10:36.473650285Z Wrote inspect data to must-gather.
2026-10-31T01:10:36.615292061Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-10-31T01:10:36.616754323Z Wrote inspect data to must-gather.
2026-10-31T01:10:36.713401316Z INFO: Getting status/flags from prometheus-k8s-0
2026-10-31T01:10:36.788940630Z INFO: LEGACY MODE
2026-10-31T01:10:36.788973861Z INFO: Gathering ovn-kubernetes master data
2026-10-31T01:10:36.816135751Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:36.850172239Z Wrote inspect data to must-gather.
2026-10-31T01:10:36.932492736Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-10-31T01:10:37.034987432Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:37.038286334Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-10-31T01:10:37.094235178Z Wrote inspect data to must-gather.
2026-10-31T01:10:37.157960510Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-10-31T01:10:37.237143306Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:37.321216433Z Wrote inspect data to must-gather.
2026-10-31T01:10:37.397109680Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-10-31T01:10:37.481974549Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:37.486673693Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-10-31T01:10:37.550431330Z Wrote inspect data to must-gather.
2026-10-31T01:10:37.595550786Z INFO: Waiting for network log collection to complete ...
2026-10-31T01:10:37.595588231Z INFO: Waiting for ovnk database copies to complete ...
2026-10-31T01:10:37.595981118Z INFO: Copying ovnk databases complete.
2026-10-31T01:10:37.597641125Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-10-31T01:10:37.598199152Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 794 is not a child of this shell
2026-10-31T01:10:37.598199152Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 795 is not a child of this shell
2026-10-31T01:10:37.598199152Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 796 is not a child of this shell
2026-10-31T01:10:37.598199152Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 798 is not a child of this shell
2026-10-31T01:10:37.598199152Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 800 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 801 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 803 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 804 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 835 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 836 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 837 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 839 is not a child of this shell
2026-10-31T01:10:37.598228302Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 840 is not a child of this shell
2026-10-31T01:10:37.599447181Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-10-31T01:10:37.602579482Z tar: Exiting with failure status due to previous errors
2026-10-31T01:10:37.603072317Z INFO: Network log collection complete.
2026-10-31T01:10:37.685341954Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:37.786114326Z Wrote inspect data to must-gather.
2026-10-31T01:10:37.789247670Z INFO: Getting status from alertmanager-main-0
2026-10-31T01:10:37.911748723Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:37.915267496Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-10-31T01:10:38.079759418Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-10-31T01:10:38.108564226Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:38.208956547Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-10-31T01:10:38.311449316Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:38.314655636Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-10-31T01:10:38.349361507Z daemonset.apps/perf-node-gather-daemonset created
2026-10-31T01:10:38.477784633Z Waiting for performance profile collector pods to become ready: 1
2026-10-31T01:10:38.552943580Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:38.667415108Z Gathering data for ns/openshift-network-console...
2026-10-31T01:10:38.748368885Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:38.758192489Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-10-31T01:10:38.931443692Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:39.125930444Z Gathering data for ns/openshift-console-operator...
2026-10-31T01:10:39.135619086Z Wrote inspect data to /must-gather/istio.
2026-10-31T01:10:39.547334179Z Gathering data for ns/openshift-console...
2026-10-31T01:10:39.603104653Z Waiting for performance profile collector pods to become ready: 2
2026-10-31T01:10:39.945445681Z Done executing Istio gather script
2026-10-31T01:10:40.097947776Z Gathering data for ns/openshift-cluster-storage-operator...
2026-10-31T01:10:40.121916695Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-10-31T01:10:40.589554918Z Gathering data for ns/openshift-dns-operator...
2026-10-31T01:10:40.726894949Z Waiting for performance profile collector pods to become ready: 3
2026-10-31T01:10:40.929439683Z Gathering data for ns/openshift-dns...
2026-10-31T01:10:41.582503393Z Gathering data for ns/openshift-image-registry...
2026-10-31T01:10:41.843552118Z Waiting for performance profile collector pods to become ready: 4
2026-10-31T01:10:42.081999533Z Gathering data for ns/openshift-ingress-operator...
2026-10-31T01:10:42.424223611Z Gathering data for ns/openshift-ingress...
2026-10-31T01:10:42.786663970Z Gathering data for ns/openshift-ingress-canary...
2026-10-31T01:10:42.968754284Z Waiting for performance profile collector pods to become ready: 5
2026-10-31T01:10:43.210448244Z Gathering data for ns/openshift-insights...
2026-10-31T01:10:44.077488355Z Waiting for performance profile collector pods to become ready: 6
2026-10-31T01:10:45.126962899Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-10-31T01:10:45.192214642Z Gathering data for ns/openshift-config...
2026-10-31T01:10:45.210278460Z Waiting for performance profile collector pods to become ready: 7
2026-10-31T01:10:45.487597307Z Gathering data for ns/openshift-config-managed...
2026-10-31T01:10:45.901006729Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-10-31T01:10:46.197828621Z Gathering data for ns/openshift-kube-apiserver...
2026-10-31T01:10:46.329122814Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-10-31T01:10:46.524072196Z Gathering data for ns/openshift-kube-controller-manager...
2026-10-31T01:10:46.609281023Z Collecting performance related data for node ip-10-0-143-150.us-east-2.compute.internal
2026-10-31T01:10:46.616050477Z Collecting performance related data for node ip-10-0-129-75.us-east-2.compute.internal
2026-10-31T01:10:46.623143746Z Collecting performance related data for node ip-10-0-135-10.us-east-2.compute.internal
2026-10-31T01:10:46.864281377Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-10-31T01:10:47.274087801Z Gathering data for ns/openshift-kube-scheduler...
2026-10-31T01:10:47.666738225Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-10-31T01:10:48.075099939Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-10-31T01:10:48.531685277Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-10-31T01:10:48.902879020Z Gathering data for ns/openshift-user-workload-monitoring...
2026-10-31T01:10:49.533476594Z Gathering data for ns/openshift-multus...
2026-10-31T01:10:50.131722593Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-10-31T01:10:51.091694061Z Gathering data for ns/openshift-ovn-kubernetes...
2026-10-31T01:10:52.454814299Z Gathering data for ns/openshift-host-network...
2026-10-31T01:10:52.785871650Z Gathering data for ns/openshift-network-diagnostics...
2026-10-31T01:10:53.352238659Z Collecting kubelet logs for node ip-10-0-143-150.us-east-2.compute.internal
2026-10-31T01:10:53.354052630Z Collecting kubelet logs for node ip-10-0-129-75.us-east-2.compute.internal
2026-10-31T01:10:53.356153561Z Collecting kubelet logs for node ip-10-0-135-10.us-east-2.compute.internal
2026-10-31T01:10:53.462120210Z Gathering data for ns/openshift-network-node-identity...
2026-10-31T01:10:53.747419892Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-kswkx namespace
2026-10-31T01:10:53.753289146Z INFO: Node performance data collection complete.
2026-10-31T01:10:53.969905190Z Gathering data for ns/openshift-network-operator...
2026-10-31T01:10:54.375023354Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-10-31T01:10:54.694647980Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-10-31T01:10:55.139051401Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-10-31T01:10:55.190178381Z Gathering data for ns/openshift-apiserver-operator...
2026-10-31T01:10:55.487729809Z Gathering data for ns/openshift-apiserver...
2026-10-31T01:10:55.864412644Z Gathering data for ns/openshift-controller-manager-operator...
2026-10-31T01:10:56.123142478Z Gathering data for ns/openshift-controller-manager...
2026-10-31T01:10:56.407497223Z Gathering data for ns/openshift-cluster-samples-operator...
2026-10-31T01:10:57.067183240Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-10-31T01:10:57.344797487Z Gathering data for ns/openshift-service-ca-operator...
2026-10-31T01:10:57.672594413Z Gathering data for ns/openshift-service-ca...
2026-10-31T01:10:58.067643786Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-10-31T01:10:59.020996540Z Wrote inspect data to must-gather.
2026-10-31T01:10:59.021045163Z error: inspection completed with the errors occurred while gathering data:
2026-10-31T01:10:59.021045163Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-10-31T01:10:59.146492138Z error: the server doesn't have a resource type "clusters"
2026-10-31T01:10:59.280050647Z Caches written to disk