Sep 17 18:27:07.061106 ip-10-0-134-202 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 17 18:27:07.061117 ip-10-0-134-202 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 17 18:27:07.061124 ip-10-0-134-202 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 17 18:27:07.061390 ip-10-0-134-202 systemd[1]: Failed to start Kubernetes Kubelet. Sep 17 18:27:17.308828 ip-10-0-134-202 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 17 18:27:17.308846 ip-10-0-134-202 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 75444f24810647758258aefee9c9f0f0 -- Sep 17 18:29:11.236825 ip-10-0-134-202 systemd[1]: Starting Kubernetes Kubelet... Sep 17 18:29:14.762218 ip-10-0-134-202 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:29:14.762218 ip-10-0-134-202 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 18:29:14.762218 ip-10-0-134-202 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:29:14.762218 ip-10-0-134-202 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 18:29:14.763071 ip-10-0-134-202 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:29:14.763071 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.762310 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 18:29:14.768427 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768402 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:14.768427 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768423 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:14.768427 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768427 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:14.768427 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768430 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:14.768427 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768433 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768436 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768440 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768458 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768462 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768466 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768469 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768472 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768474 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768477 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768480 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768483 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768487 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768489 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768492 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768495 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768503 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768505 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768508 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768511 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:14.768644 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768514 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768516 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768519 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768521 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768524 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768527 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768531 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768533 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768538 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768542 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768545 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768548 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768551 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768554 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768557 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768560 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768563 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768566 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768569 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:14.769112 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768572 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768575 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768577 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768580 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768582 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768586 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768589 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768591 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768594 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768597 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768599 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768602 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768604 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768607 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768610 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768613 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768615 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768619 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768623 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768625 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:14.769601 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768629 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768632 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768634 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768637 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768640 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768643 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768646 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768650 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768653 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768656 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768659 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768662 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768665 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768667 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768670 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768673 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768676 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768679 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768681 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:14.770081 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768684 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768692 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768695 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768698 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768700 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768703 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768706 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768709 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768711 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768714 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768716 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768719 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768722 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768724 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768727 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768729 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.768732 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769475 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769482 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769486 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:14.770546 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769491 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769495 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769498 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769502 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769505 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769508 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769511 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769513 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769516 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769519 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769522 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769525 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769528 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769530 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769534 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769536 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769539 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769542 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769545 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:14.771035 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769548 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769551 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769554 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769556 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769559 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769562 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769564 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769567 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769570 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769572 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769575 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769578 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769581 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769583 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769586 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769589 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769591 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769594 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769596 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769599 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:14.771525 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769602 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769604 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769607 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769610 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769612 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769616 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769618 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769621 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769624 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769627 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769630 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769633 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769635 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769638 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769641 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769643 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769646 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769649 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769651 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769653 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:14.772026 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769656 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769658 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769661 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769663 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769666 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769668 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769671 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769676 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769680 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769683 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769686 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769689 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769692 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769694 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769697 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769699 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769702 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769704 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769707 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:14.772531 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769709 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769711 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769715 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769718 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769721 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769724 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769726 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769729 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769732 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769734 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769737 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769739 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769743 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769747 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769750 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769752 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769755 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.769757 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770151 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770156 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:14.772989 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770159 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770167 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770171 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770173 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770176 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770179 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770182 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770185 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770187 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770190 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770193 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770195 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770198 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770201 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770204 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770207 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770209 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770212 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770215 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770217 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:14.773534 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770220 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770222 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770225 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770227 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770230 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770233 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770235 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770239 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770243 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770246 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770248 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770251 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770253 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770258 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770260 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770263 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770265 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770269 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770273 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:14.774037 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770276 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770279 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770282 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770285 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770288 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770290 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770294 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770297 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770300 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770303 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770306 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770309 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770311 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770314 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770317 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770320 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770322 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770325 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770327 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770330 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:14.774506 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770332 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770335 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770337 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770340 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770342 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770345 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770349 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770352 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770356 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770359 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770362 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770365 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770368 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770370 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770373 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770375 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770378 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770380 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770383 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770386 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:14.774984 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770388 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770391 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770393 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770396 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770401 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770403 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770406 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770408 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770411 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770414 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770416 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770419 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770421 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770424 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770426 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770429 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770431 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.770437 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770528 2568 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770566 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770573 2568 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 18:29:14.775535 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770578 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770582 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770585 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770590 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770595 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770598 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770601 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770604 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770608 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770611 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770614 2568 flags.go:64] FLAG: --cgroup-root="" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770617 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770619 2568 flags.go:64] FLAG: --client-ca-file="" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770623 2568 flags.go:64] FLAG: --cloud-provider="external" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770626 2568 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770631 2568 flags.go:64] FLAG: --cluster-domain="" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770633 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770637 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770640 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770643 2568 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770647 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770650 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770653 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 18:29:14.776039 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770657 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770660 2568 flags.go:64] FLAG: --contention-profiling="false" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770663 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770666 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770669 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770674 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770679 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770682 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770685 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770688 2568 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770691 2568 flags.go:64] FLAG: --enable-server="true" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770694 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770698 2568 flags.go:64] FLAG: --event-burst="100" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770701 2568 flags.go:64] FLAG: --event-qps="50" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770704 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770707 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770710 2568 flags.go:64] FLAG: --eviction-hard="" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770713 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770716 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770719 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770722 2568 flags.go:64] FLAG: --eviction-soft="" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770725 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770728 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770731 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770734 2568 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 18:29:14.776624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770737 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770740 2568 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770743 2568 flags.go:64] FLAG: --feature-gates="" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770747 2568 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770750 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770753 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770757 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770759 2568 flags.go:64] FLAG: --healthz-port="10248" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770763 2568 flags.go:64] FLAG: --help="false" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770765 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-134-202.ec2.internal" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770768 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770771 2568 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770776 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770779 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770783 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770786 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770789 2568 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770792 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770795 2568 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770798 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770801 2568 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770804 2568 flags.go:64] FLAG: --kube-reserved="" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770806 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770810 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 18:29:14.777248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770813 2568 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770815 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770818 2568 flags.go:64] FLAG: --lock-file="" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770821 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770824 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770827 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770833 2568 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770837 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770840 2568 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770843 2568 flags.go:64] FLAG: --logging-format="text" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770847 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770850 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770853 2568 flags.go:64] FLAG: --manifest-url="" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770856 2568 flags.go:64] FLAG: --manifest-url-header="" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770861 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770864 2568 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770872 2568 flags.go:64] FLAG: --max-pods="110" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770875 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770878 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770881 2568 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770885 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770888 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770891 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770894 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770902 2568 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 18:29:14.777866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770905 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770908 2568 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770911 2568 flags.go:64] FLAG: --pod-cidr="" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770913 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770920 2568 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770923 2568 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770926 2568 flags.go:64] FLAG: --pods-per-core="0" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770929 2568 flags.go:64] FLAG: --port="10250" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770932 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770935 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b8181a1aa0587649" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770938 2568 flags.go:64] FLAG: --qos-reserved="" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770941 2568 flags.go:64] FLAG: --read-only-port="10255" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770944 2568 flags.go:64] FLAG: --register-node="true" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770947 2568 flags.go:64] FLAG: --register-with-taints="" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770950 2568 flags.go:64] FLAG: --registry-burst="10" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770954 2568 flags.go:64] FLAG: --registry-qps="5" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770957 2568 flags.go:64] FLAG: --reserved-cpus="" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770960 2568 flags.go:64] FLAG: --reserved-memory="" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770963 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770966 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770969 2568 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770972 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770975 2568 flags.go:64] FLAG: --runonce="false" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770978 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770981 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 18:29:14.778487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770983 2568 flags.go:64] FLAG: --seccomp-default="false" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770986 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770989 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770994 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.770997 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771001 2568 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771003 2568 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771006 2568 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771009 2568 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771012 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771015 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771018 2568 flags.go:64] FLAG: --system-cgroups="" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771021 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771026 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771028 2568 flags.go:64] FLAG: --tls-cert-file="" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771031 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771036 2568 flags.go:64] FLAG: --tls-min-version="" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771039 2568 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771042 2568 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771045 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771048 2568 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771052 2568 flags.go:64] FLAG: --v="2" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771056 2568 flags.go:64] FLAG: --version="false" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771061 2568 flags.go:64] FLAG: --vmodule="" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771065 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 18:29:14.779088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771068 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771202 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771206 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771209 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771212 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771215 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771218 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771220 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771223 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771226 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771231 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771235 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771237 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771240 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771243 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771246 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771248 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771252 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771256 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771259 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:14.779713 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771262 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771265 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771268 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771271 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771274 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771277 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771280 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771283 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771287 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771290 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771294 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771304 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771307 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771310 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771313 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771315 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771318 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771321 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771323 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:14.780250 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771326 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771328 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771331 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771337 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771339 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771342 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771345 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771349 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771352 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771355 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771358 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771362 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771364 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771367 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771370 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771373 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771375 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771378 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771380 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:14.780721 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771383 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771386 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771389 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771391 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771394 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771396 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771403 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771406 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771408 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771411 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771413 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771416 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771419 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771421 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771424 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771427 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771432 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771436 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771438 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:14.781181 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771441 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771455 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771458 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771460 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771463 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771466 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771468 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771471 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771474 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771477 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771479 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771482 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771484 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771487 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771490 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771493 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771495 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771498 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771501 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771503 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:14.781657 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771506 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771509 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.771512 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.771517 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.778398 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.778415 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.778437 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.778461 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778523 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778528 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778531 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778535 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778538 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778541 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778544 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:14.782148 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778547 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778550 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778553 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778555 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778558 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778561 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778564 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778566 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778569 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778572 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778575 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778577 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778580 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778582 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778585 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778588 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778590 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778593 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778595 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778598 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:14.782540 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778601 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778605 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778614 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778618 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778620 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778623 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778626 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778628 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778631 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778633 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778636 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778639 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778642 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778644 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778647 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778649 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778652 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778655 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778657 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:14.783022 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778660 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778664 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778667 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778670 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778673 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778677 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778679 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778682 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778685 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778687 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778690 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778693 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778695 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778698 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778700 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778704 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778714 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778718 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778721 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:14.783503 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778723 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778726 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778729 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778731 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778734 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778737 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778739 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778742 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778745 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778747 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778750 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778753 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778755 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778758 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778760 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778763 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778765 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778768 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778770 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778773 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:14.783990 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778775 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778778 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778780 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778783 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778785 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778788 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778791 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778794 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778797 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778799 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778808 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778810 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778813 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778816 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.778821 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:29:14.784489 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778982 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778987 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778990 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778993 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.778997 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779000 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779003 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779005 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779008 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779011 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779014 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779017 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779019 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779022 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779025 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779027 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779030 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779033 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779035 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:29:14.784884 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779040 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779044 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779047 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779049 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779052 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779055 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779058 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779061 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779064 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779072 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779075 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779078 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779081 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779083 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779086 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779088 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779091 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779094 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779097 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:29:14.785341 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779099 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779102 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779105 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779107 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779110 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779112 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779115 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779117 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779120 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779122 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779125 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779127 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779131 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779135 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779139 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779142 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779145 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779148 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779151 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:29:14.785824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779153 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779157 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779160 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779163 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779171 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779175 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779177 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779181 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779183 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779186 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779188 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779191 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779194 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779196 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779199 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779201 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779204 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779206 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779209 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779211 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:29:14.786293 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779214 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779217 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779219 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779222 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779224 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779227 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779230 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779232 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779235 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779237 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779239 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779242 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779245 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779247 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779250 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779254 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779257 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779265 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779268 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779271 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:29:14.787053 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779273 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:14.779276 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.779280 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.779441 2568 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.781296 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.782222 2568 server.go:1019] "Starting client certificate rotation" Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.782349 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.782383 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:29:14.787824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.784610 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:29:14.797232 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.797205 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:29:14.807715 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.807697 2568 log.go:25] "Validated CRI v1 runtime API" Sep 17 18:29:14.809506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.809490 2568 log.go:25] "Validated CRI v1 image API" Sep 17 18:29:14.810351 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.810325 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 18:29:14.813068 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.813044 2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad3782b5-1f28-4b99-9755-1811e0c033dd:/dev/nvme0n1p4 d4b92b09-2f90-4664-97eb-3d49e4742de2:/dev/nvme0n1p3] Sep 17 18:29:14.813158 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.813067 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 18:29:14.817609 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.817480 2568 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:14.816980246 +0000 UTC m=+2.096631611 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099890 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2970b6c17d439bc4b1f9f82a18bcfd SystemUUID:ec2970b6-c17d-439b-c4b1-f9f82a18bcfd BootID:75444f24-8106-4775-8258-aefee9c9f0f0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:3e:6e:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:3e:6e:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:12:98:e4:9a:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 18:29:14.817609 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.817598 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 18:29:14.817792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.817723 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 18:29:14.819740 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.819703 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 18:29:14.819904 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.819744 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 18:29:14.820860 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.820847 2568 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 18:29:14.820914 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.820862 2568 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 18:29:14.820914 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.820897 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:29:14.820914 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.820913 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:29:14.820997 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.820925 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:29:14.821028 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.821008 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 18:29:14.823613 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.823600 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:29:14.823740 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.823722 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 18:29:14.823774 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.823761 2568 kubelet.go:486] "Attempting to sync node with API server" Sep 17 18:29:14.823774 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.823773 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 18:29:14.823829 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.823789 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 18:29:14.823829 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.823798 2568 kubelet.go:398] "Adding apiserver pod source" Sep 17 18:29:14.823829 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.823808 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 18:29:14.824263 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.824249 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:29:14.824310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.824286 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:29:14.828693 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.828669 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 18:29:14.830935 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.830921 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 18:29:14.830986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.830951 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 18:29:14.832225 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832203 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832234 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832245 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832253 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832273 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832281 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832300 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832309 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 18:29:14.832320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832319 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 18:29:14.832667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832328 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 18:29:14.832667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832341 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 18:29:14.832667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832349 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 18:29:14.832667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832382 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 18:29:14.832667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.832389 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 18:29:14.836647 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.836620 2568 server.go:1268] "Started kubelet" Sep 17 18:29:14.836749 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.836664 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 18:29:14.836806 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.836728 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 18:29:14.836806 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.836793 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 18:29:14.837534 ip-10-0-134-202 systemd[1]: Started Kubernetes Kubelet. Sep 17 18:29:14.838135 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.837712 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 18:29:14.838135 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.837929 2568 server.go:310] "Adding debug handlers to kubelet server" Sep 17 18:29:14.839041 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.839019 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-202.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 18:29:14.839361 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.839343 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 18:29:14.839361 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.839344 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 18:29:14.841875 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.841854 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 18:29:14.843528 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.843513 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 18:29:14.843598 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.843537 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 18:29:14.843782 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.843755 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:14.844103 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844077 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 18:29:14.844192 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844157 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 18:29:14.844192 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844172 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 18:29:14.844425 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844404 2568 reconstruct.go:97] "Volume reconstruction finished" Sep 17 18:29:14.844425 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844427 2568 reconciler.go:29] "Reconciler: start to sync state" Sep 17 18:29:14.844643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844628 2568 factory.go:55] Registering systemd factory Sep 17 18:29:14.844688 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844649 2568 factory.go:223] Registration of the systemd container factory successfully Sep 17 18:29:14.844926 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844912 2568 factory.go:153] Registering CRI-O factory Sep 17 18:29:14.844926 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.844928 2568 factory.go:223] Registration of the crio container factory successfully Sep 17 18:29:14.845033 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.845011 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 18:29:14.845033 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.845027 2568 factory.go:103] Registering Raw factory Sep 17 18:29:14.845144 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.845043 2568 manager.go:1196] Started watching for new ooms in manager Sep 17 18:29:14.846350 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.846326 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 18:29:14.846507 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.846486 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 18:29:14.846583 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.846513 2568 manager.go:349] Starting recovery of all containers Sep 17 18:29:14.870985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.870964 2568 manager.go:354] Recovery completed Sep 17 18:29:14.872192 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.871201 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-202.ec2.internal.18d62f28f2488d4e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-202.ec2.internal,UID:ip-10-0-134-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-202.ec2.internal,},FirstTimestamp:2026-09-17 18:29:14.836389198 +0000 UTC m=+2.116040563,LastTimestamp:2026-09-17 18:29:14.836389198 +0000 UTC m=+2.116040563,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-202.ec2.internal,}" Sep 17 18:29:14.874090 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.874073 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b2b6w" Sep 17 18:29:14.875409 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.875397 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:14.880034 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.880017 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b2b6w" Sep 17 18:29:14.882755 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.882734 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:14.882830 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.882765 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:14.882830 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.882775 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:14.883479 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.883458 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 18:29:14.883479 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.883474 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:14.883567 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.883479 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 18:29:14.883567 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.883566 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:29:14.884256 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.884241 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:14.884321 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.884269 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:14.884321 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.884280 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:14.884707 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.884693 2568 policy_none.go:49] "None policy: Start" Sep 17 18:29:14.884765 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.884710 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 18:29:14.884765 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.884720 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 18:29:14.885822 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.885810 2568 policy_none.go:47] "Start" Sep 17 18:29:14.934167 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.934147 2568 manager.go:337] "Starting Device Plugin manager" Sep 17 18:29:14.934291 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.934216 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 18:29:14.934291 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.934227 2568 server.go:87] "Starting device plugin registration server" Sep 17 18:29:14.934552 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.934538 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 18:29:14.934631 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.934564 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 18:29:14.934684 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.934663 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 18:29:14.934776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.934762 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 18:29:14.934829 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.934776 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 18:29:14.935316 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.935293 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 18:29:14.935409 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.935356 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:14.977040 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.977000 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 18:29:14.978184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.978168 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 18:29:14.978244 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.978222 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 18:29:14.978295 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.978248 2568 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 18:29:14.978347 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:14.978303 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 18:29:14.980729 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:14.980709 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:15.034749 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.034662 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:15.035786 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.035768 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:15.035904 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.035803 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:15.035904 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.035820 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:15.035904 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.035850 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.045569 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.045542 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.045696 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.045575 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-202.ec2.internal\": node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.074181 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.074147 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.079197 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.079174 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"] Sep 17 18:29:15.079296 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.079276 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:15.080859 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.080844 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:15.080938 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.080873 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:15.080938 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.080888 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:15.081986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.081974 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:15.082143 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082129 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.082186 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082160 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:15.082811 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082796 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:15.082811 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082808 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:15.082911 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082829 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:15.082911 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082831 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:15.082911 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082849 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:15.082911 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.082840 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:15.084748 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.084732 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.084799 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.084758 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:29:15.085513 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.085498 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:29:15.085581 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.085525 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:29:15.085581 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.085534 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:29:15.110299 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.110267 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-202.ec2.internal\" not found" node="ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.114981 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.114960 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-202.ec2.internal\" not found" node="ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.146623 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.146585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4098de4361d529da9156a4482a8f5692-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"4098de4361d529da9156a4482a8f5692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.146623 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.146617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4098de4361d529da9156a4482a8f5692-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"4098de4361d529da9156a4482a8f5692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.146853 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.146669 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f316321b0ddd050eb379ecf12c03828-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"1f316321b0ddd050eb379ecf12c03828\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.174546 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.174516 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.247132 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.247092 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4098de4361d529da9156a4482a8f5692-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"4098de4361d529da9156a4482a8f5692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.247307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.247143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4098de4361d529da9156a4482a8f5692-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"4098de4361d529da9156a4482a8f5692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.247307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.247194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f316321b0ddd050eb379ecf12c03828-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"1f316321b0ddd050eb379ecf12c03828\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.247307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.247209 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4098de4361d529da9156a4482a8f5692-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"4098de4361d529da9156a4482a8f5692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.247307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.247226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f316321b0ddd050eb379ecf12c03828-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"1f316321b0ddd050eb379ecf12c03828\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.247307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.247257 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4098de4361d529da9156a4482a8f5692-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"4098de4361d529da9156a4482a8f5692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.275250 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.275214 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.376034 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.375963 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.413152 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.413125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.418487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.418467 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Sep 17 18:29:15.476817 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.476638 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.577148 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.577117 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.677731 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.677656 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.778203 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.778162 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.782340 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.782319 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 18:29:15.782523 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.782506 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 18:29:15.878368 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.878318 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.885168 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.885131 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:14 +0000 UTC" deadline="2028-02-18 01:41:34.774800874 +0000 UTC" Sep 17 18:29:15.885168 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:15.885162 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12439h12m18.889642903s" Sep 17 18:29:15.978905 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:15.978869 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Sep 17 18:29:15.995156 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:15.995123 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f316321b0ddd050eb379ecf12c03828.slice/crio-8ded79ffddba50db1231b9e0ee2d371e5258ddd668e12bcc17e620c16c44e498 WatchSource:0}: Error finding container 8ded79ffddba50db1231b9e0ee2d371e5258ddd668e12bcc17e620c16c44e498: Status 404 returned error can't find the container with id 8ded79ffddba50db1231b9e0ee2d371e5258ddd668e12bcc17e620c16c44e498 Sep 17 18:29:15.995637 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:15.995611 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4098de4361d529da9156a4482a8f5692.slice/crio-72e5643395bb18ddd7724443d4c54c1aab5f9201167f4ad73fa630230d6151cc WatchSource:0}: Error finding container 72e5643395bb18ddd7724443d4c54c1aab5f9201167f4ad73fa630230d6151cc: Status 404 returned error can't find the container with id 72e5643395bb18ddd7724443d4c54c1aab5f9201167f4ad73fa630230d6151cc Sep 17 18:29:16.002067 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.002047 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:29:16.034280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.034246 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:16.044079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.044047 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Sep 17 18:29:16.052957 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.052931 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:16.058848 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.058826 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:29:16.060346 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.060331 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Sep 17 18:29:16.068874 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.068859 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:29:16.197835 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.197811 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:16.824822 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.824743 2568 apiserver.go:52] "Watching apiserver" Sep 17 18:29:16.831585 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.831562 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 18:29:16.832018 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.831992 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-vhft5","openshift-multus/multus-additional-cni-plugins-tbgsj","openshift-network-diagnostics/network-check-target-wh8zt","openshift-network-operator/iptables-alerter-mrpkj","kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal","openshift-dns/node-resolver-75d5b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal","openshift-multus/multus-r82qp","openshift-multus/network-metrics-daemon-zq6bz","openshift-ovn-kubernetes/ovnkube-node-895jp","kube-system/global-pull-secret-syncer-6hvl9","kube-system/konnectivity-agent-sbdzm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f","openshift-cluster-node-tuning-operator/tuned-mvqf7"] Sep 17 18:29:16.833830 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.833808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.835433 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.835257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.835433 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.835328 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:16.836388 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.836365 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 18:29:16.837254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.836990 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 18:29:16.837254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.836990 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 18:29:16.837254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.837033 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 18:29:16.837254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.837093 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 18:29:16.837254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.837035 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tmvt\"" Sep 17 18:29:16.837558 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.837315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:16.837558 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.837403 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 18:29:16.837558 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.837433 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:16.839231 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.838917 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.841001 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.840978 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:16.841271 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.841252 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s6thp\"" Sep 17 18:29:16.841523 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.841507 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:16.842516 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.841677 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 18:29:16.844177 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.844154 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.844727 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.844593 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 18:29:16.845180 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.845158 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.846474 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.846425 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:16.846563 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.846513 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:16.846622 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.846602 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:16.847262 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.847122 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v6hp4\"" Sep 17 18:29:16.847262 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.847175 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:16.847762 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.847623 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 18:29:16.847762 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.847651 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 18:29:16.847762 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.847658 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ss8rl\"" Sep 17 18:29:16.848557 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.848538 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.850344 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.850295 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.851504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.850832 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 18:29:16.851504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.850917 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmtgv\"" Sep 17 18:29:16.851504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.850835 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 18:29:16.851504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.851148 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 18:29:16.852007 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.851644 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:16.853030 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.853012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.853390 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.853372 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 18:29:16.853792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.853776 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 18:29:16.853878 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.853843 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 18:29:16.854088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854052 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g766x\"" Sep 17 18:29:16.854165 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854129 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 18:29:16.854212 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854177 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bwktv\"" Sep 17 18:29:16.854272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854129 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 18:29:16.854396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854372 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:29:16.854476 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854403 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 18:29:16.854476 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854461 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 18:29:16.854606 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854583 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 18:29:16.854972 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.854950 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qnzd9\"" Sep 17 18:29:16.855682 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.855661 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:16.855787 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.855670 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 18:29:16.857059 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-sys\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.857188 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-var-lib-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857188 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857087 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-ovn\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857188 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-log-socket\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857188 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857131 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857188 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-env-overrides\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52d928d4-4882-46ee-91c9-47e723e32b3a-ovn-node-metrics-cert\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-ovnkube-script-lib\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857248 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-run-netns\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-systemd\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566hq\" (UniqueName: \"kubernetes.io/projected/01a5d8fc-8768-4c7d-971a-485e2cd09de7-kube-api-access-566hq\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysctl-conf\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00ff0fa7-5e71-45db-a096-722ef629cfca-tmp\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.857418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-systemd-units\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857507 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-etc-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-run\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcsk\" (UniqueName: \"kubernetes.io/projected/c672c626-1477-4611-a266-16e533c89d87-kube-api-access-xqcsk\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8747e254-9790-4236-87a0-9955cb416dde-dbus\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-systemd\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01a5d8fc-8768-4c7d-971a-485e2cd09de7-host-slash\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-modprobe-d\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.857805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857796 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-tuned\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857871 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-node-log\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857940 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysconfig\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857973 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-lib-modules\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857878 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9shpq\"" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.857995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-cni-bin\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858039 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgc4z\" (UniqueName: \"kubernetes.io/projected/00ff0fa7-5e71-45db-a096-722ef629cfca-kube-api-access-jgc4z\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-kubelet\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-slash\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858114 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-ovnkube-config\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8747e254-9790-4236-87a0-9955cb416dde-kubelet-config\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.858175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858202 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/01a5d8fc-8768-4c7d-971a-485e2cd09de7-iptables-alerter-script\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-kubernetes\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858249 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c672c626-1477-4611-a266-16e533c89d87-hosts-file\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kw65\" (UniqueName: \"kubernetes.io/projected/52d928d4-4882-46ee-91c9-47e723e32b3a-kube-api-access-2kw65\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858295 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858341 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysctl-d\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-var-lib-kubelet\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-host\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c672c626-1477-4611-a266-16e533c89d87-tmp-dir\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.858710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.858458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-cni-netd\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.876196 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.876163 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdpmq" Sep 17 18:29:16.885082 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.885057 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdpmq" Sep 17 18:29:16.945405 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.945372 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 18:29:16.958724 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566hq\" (UniqueName: \"kubernetes.io/projected/01a5d8fc-8768-4c7d-971a-485e2cd09de7-kube-api-access-566hq\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.958724 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysctl-conf\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.958954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958754 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00ff0fa7-5e71-45db-a096-722ef629cfca-tmp\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.958954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-etc-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.958954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.958954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958910 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysctl-conf\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.958954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.958954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-run\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8747e254-9790-4236-87a0-9955cb416dde-dbus\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958966 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.958996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-sys-fs\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959027 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-etc-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01a5d8fc-8768-4c7d-971a-485e2cd09de7-host-slash\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-modprobe-d\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-node-log\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959117 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959136 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-daemon-config\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01a5d8fc-8768-4c7d-971a-485e2cd09de7-host-slash\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqs48\" (UniqueName: \"kubernetes.io/projected/2b74e7b8-7738-4fc8-9b08-27179c470990-kube-api-access-mqs48\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959177 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8747e254-9790-4236-87a0-9955cb416dde-dbus\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysconfig\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959192 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-node-log\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959183 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-run\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.959238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-lib-modules\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959245 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysconfig\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97x6n\" (UniqueName: \"kubernetes.io/projected/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-kube-api-access-97x6n\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959257 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-modprobe-d\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-socket-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-registration-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgc4z\" (UniqueName: \"kubernetes.io/projected/00ff0fa7-5e71-45db-a096-722ef629cfca-kube-api-access-jgc4z\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-lib-modules\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-kubelet\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-slash\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959409 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-ovnkube-config\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959432 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8747e254-9790-4236-87a0-9955cb416dde-kubelet-config\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959482 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-slash\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-kubelet\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959552 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8747e254-9790-4236-87a0-9955cb416dde-kubelet-config\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:16.960111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/092bdb7d-656a-4900-8af6-f0a03eab5aa9-host\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.959595 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2sn\" (UniqueName: \"kubernetes.io/projected/092bdb7d-656a-4900-8af6-f0a03eab5aa9-kube-api-access-7q2sn\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-k8s-cni-cncf-io\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959635 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-cni-multus\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5pn\" (UniqueName: \"kubernetes.io/projected/e3d8596e-c089-4b0b-89ab-68002f3c78d8-kube-api-access-tl5pn\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.959689 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret podName:8747e254-9790-4236-87a0-9955cb416dde nodeName:}" failed. No retries permitted until 2026-09-17 18:29:17.459651002 +0000 UTC m=+4.739302357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret") pod "global-pull-secret-syncer-6hvl9" (UID: "8747e254-9790-4236-87a0-9955cb416dde") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959776 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-device-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-cnibin\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959888 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysctl-d\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-host\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959969 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-cni-netd\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.959993 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-netns\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960005 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-ovnkube-config\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.960902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-kubelet\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960034 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-sysctl-d\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960047 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kw65\" (UniqueName: \"kubernetes.io/projected/52d928d4-4882-46ee-91c9-47e723e32b3a-kube-api-access-2kw65\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960059 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-cni-netd\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-sys\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960086 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-host\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960109 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-var-lib-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960117 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-sys\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-env-overrides\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960155 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-var-lib-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960161 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-cnibin\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-run-netns\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-systemd-units\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-conf-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-run-netns\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-systemd-units\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960268 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.961672 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcsk\" (UniqueName: \"kubernetes.io/projected/c672c626-1477-4611-a266-16e533c89d87-kube-api-access-xqcsk\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960356 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-socket-dir-parent\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960383 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-multus-certs\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d63869ab-2129-4c22-bdde-754ea60a7cf6-konnectivity-ca\") pod \"konnectivity-agent-sbdzm\" (UID: \"d63869ab-2129-4c22-bdde-754ea60a7cf6\") " pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960465 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-system-cni-dir\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-systemd\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-tuned\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960568 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-env-overrides\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-systemd\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-cni-bin\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960599 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-openvswitch\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960612 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-system-cni-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960641 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-cni-bin\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960640 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzfn\" (UniqueName: \"kubernetes.io/projected/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-kube-api-access-gfzfn\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-etc-kubernetes\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d63869ab-2129-4c22-bdde-754ea60a7cf6-agent-certs\") pod \"konnectivity-agent-sbdzm\" (UID: \"d63869ab-2129-4c22-bdde-754ea60a7cf6\") " pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:16.962272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960746 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/01a5d8fc-8768-4c7d-971a-485e2cd09de7-iptables-alerter-script\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-kubernetes\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c672c626-1477-4611-a266-16e533c89d87-hosts-file\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960832 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960850 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-kubernetes\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/092bdb7d-656a-4900-8af6-f0a03eab5aa9-serviceca\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960876 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c672c626-1477-4611-a266-16e533c89d87-hosts-file\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960895 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-cni-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-hostroot\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-var-lib-kubelet\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c672c626-1477-4611-a266-16e533c89d87-tmp-dir\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.960991 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00ff0fa7-5e71-45db-a096-722ef629cfca-var-lib-kubelet\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-cni-bin\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961046 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-ovn\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-log-socket\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961093 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961103 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-ovn\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52d928d4-4882-46ee-91c9-47e723e32b3a-ovn-node-metrics-cert\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-ovnkube-script-lib\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961152 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-log-socket\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961176 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d8596e-c089-4b0b-89ab-68002f3c78d8-cni-binary-copy\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-systemd\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961266 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c672c626-1477-4611-a266-16e533c89d87-tmp-dir\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961301 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52d928d4-4882-46ee-91c9-47e723e32b3a-run-systemd\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961435 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/01a5d8fc-8768-4c7d-971a-485e2cd09de7-iptables-alerter-script\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.961839 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52d928d4-4882-46ee-91c9-47e723e32b3a-ovnkube-script-lib\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.963038 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00ff0fa7-5e71-45db-a096-722ef629cfca-tmp\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.963225 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00ff0fa7-5e71-45db-a096-722ef629cfca-etc-tuned\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.963880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.963469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52d928d4-4882-46ee-91c9-47e723e32b3a-ovn-node-metrics-cert\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.967010 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.966987 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgc4z\" (UniqueName: \"kubernetes.io/projected/00ff0fa7-5e71-45db-a096-722ef629cfca-kube-api-access-jgc4z\") pod \"tuned-mvqf7\" (UID: \"00ff0fa7-5e71-45db-a096-722ef629cfca\") " pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:16.967255 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.967238 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-566hq\" (UniqueName: \"kubernetes.io/projected/01a5d8fc-8768-4c7d-971a-485e2cd09de7-kube-api-access-566hq\") pod \"iptables-alerter-mrpkj\" (UID: \"01a5d8fc-8768-4c7d-971a-485e2cd09de7\") " pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:16.972124 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.972098 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kw65\" (UniqueName: \"kubernetes.io/projected/52d928d4-4882-46ee-91c9-47e723e32b3a-kube-api-access-2kw65\") pod \"ovnkube-node-895jp\" (UID: \"52d928d4-4882-46ee-91c9-47e723e32b3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:16.972847 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.972815 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:16.972847 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.972846 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:16.973013 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.972860 2568 projected.go:196] Error preparing data for projected volume kube-api-access-mb4hp for pod openshift-network-diagnostics/network-check-target-wh8zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:16.973013 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:16.972948 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp podName:00162636-5bc2-435a-8fbe-39f28f63641d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:17.472920563 +0000 UTC m=+4.752571932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mb4hp" (UniqueName: "kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp") pod "network-check-target-wh8zt" (UID: "00162636-5bc2-435a-8fbe-39f28f63641d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:16.974641 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.974624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcsk\" (UniqueName: \"kubernetes.io/projected/c672c626-1477-4611-a266-16e533c89d87-kube-api-access-xqcsk\") pod \"node-resolver-75d5b\" (UID: \"c672c626-1477-4611-a266-16e533c89d87\") " pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:16.983306 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.983258 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"4098de4361d529da9156a4482a8f5692","Type":"ContainerStarted","Data":"72e5643395bb18ddd7724443d4c54c1aab5f9201167f4ad73fa630230d6151cc"} Sep 17 18:29:16.984573 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:16.984543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" event={"ID":"1f316321b0ddd050eb379ecf12c03828","Type":"ContainerStarted","Data":"8ded79ffddba50db1231b9e0ee2d371e5258ddd668e12bcc17e620c16c44e498"} Sep 17 18:29:17.062307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/092bdb7d-656a-4900-8af6-f0a03eab5aa9-serviceca\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:17.062307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062303 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-cni-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062546 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-hostroot\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062546 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062408 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-cni-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062546 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062427 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-cni-bin\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062546 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.062546 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062496 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-hostroot\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062546 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d8596e-c089-4b0b-89ab-68002f3c78d8-cni-binary-copy\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062546 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-sys-fs\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062626 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-daemon-config\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqs48\" (UniqueName: \"kubernetes.io/projected/2b74e7b8-7738-4fc8-9b08-27179c470990-kube-api-access-mqs48\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062655 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-sys-fs\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062560 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-cni-bin\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.062764 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.062821 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs podName:370e5c7b-6a22-41a2-bbe8-67e7f11bc373 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:17.562803464 +0000 UTC m=+4.842454839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs") pod "network-metrics-daemon-zq6bz" (UID: "370e5c7b-6a22-41a2-bbe8-67e7f11bc373") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:17.062894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.062835 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/092bdb7d-656a-4900-8af6-f0a03eab5aa9-serviceca\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063110 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-daemon-config\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97x6n\" (UniqueName: \"kubernetes.io/projected/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-kube-api-access-97x6n\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d8596e-c089-4b0b-89ab-68002f3c78d8-cni-binary-copy\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-socket-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063156 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063179 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-registration-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063223 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/092bdb7d-656a-4900-8af6-f0a03eab5aa9-host\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-socket-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063247 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2sn\" (UniqueName: \"kubernetes.io/projected/092bdb7d-656a-4900-8af6-f0a03eab5aa9-kube-api-access-7q2sn\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063279 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-registration-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-k8s-cni-cncf-io\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063314 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/092bdb7d-656a-4900-8af6-f0a03eab5aa9-host\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-cni-multus\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063352 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-cni-multus\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063357 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5pn\" (UniqueName: \"kubernetes.io/projected/e3d8596e-c089-4b0b-89ab-68002f3c78d8-kube-api-access-tl5pn\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.063396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-k8s-cni-cncf-io\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063406 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-device-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-cnibin\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063501 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-netns\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063520 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-device-dir\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-cnibin\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063556 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-netns\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-kubelet\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-cnibin\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063635 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-conf-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063652 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-var-lib-kubelet\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063659 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063699 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-socket-dir-parent\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-multus-certs\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d63869ab-2129-4c22-bdde-754ea60a7cf6-konnectivity-ca\") pod \"konnectivity-agent-sbdzm\" (UID: \"d63869ab-2129-4c22-bdde-754ea60a7cf6\") " pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-system-cni-dir\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063799 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-system-cni-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzfn\" (UniqueName: \"kubernetes.io/projected/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-kube-api-access-gfzfn\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063858 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-cnibin\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-etc-kubernetes\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063901 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-conf-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063907 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-multus-socket-dir-parent\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063893 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d63869ab-2129-4c22-bdde-754ea60a7cf6-agent-certs\") pod \"konnectivity-agent-sbdzm\" (UID: \"d63869ab-2129-4c22-bdde-754ea60a7cf6\") " pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063951 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b74e7b8-7738-4fc8-9b08-27179c470990-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063955 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-system-cni-dir\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063981 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-host-run-multus-certs\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.063988 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b74e7b8-7738-4fc8-9b08-27179c470990-system-cni-dir\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.064031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d8596e-c089-4b0b-89ab-68002f3c78d8-etc-kubernetes\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.064921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.064541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d63869ab-2129-4c22-bdde-754ea60a7cf6-konnectivity-ca\") pod \"konnectivity-agent-sbdzm\" (UID: \"d63869ab-2129-4c22-bdde-754ea60a7cf6\") " pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:17.066580 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.066560 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d63869ab-2129-4c22-bdde-754ea60a7cf6-agent-certs\") pod \"konnectivity-agent-sbdzm\" (UID: \"d63869ab-2129-4c22-bdde-754ea60a7cf6\") " pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:17.096106 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.096025 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5pn\" (UniqueName: \"kubernetes.io/projected/e3d8596e-c089-4b0b-89ab-68002f3c78d8-kube-api-access-tl5pn\") pod \"multus-r82qp\" (UID: \"e3d8596e-c089-4b0b-89ab-68002f3c78d8\") " pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.096261 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.096170 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97x6n\" (UniqueName: \"kubernetes.io/projected/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-kube-api-access-97x6n\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:17.096261 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.096170 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzfn\" (UniqueName: \"kubernetes.io/projected/f8ab30d6-ddf1-4b86-9c27-53625f671ccc-kube-api-access-gfzfn\") pod \"aws-ebs-csi-driver-node-nmn6f\" (UID: \"f8ab30d6-ddf1-4b86-9c27-53625f671ccc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.096462 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.096426 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2sn\" (UniqueName: \"kubernetes.io/projected/092bdb7d-656a-4900-8af6-f0a03eab5aa9-kube-api-access-7q2sn\") pod \"node-ca-vhft5\" (UID: \"092bdb7d-656a-4900-8af6-f0a03eab5aa9\") " pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:17.097133 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.097111 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqs48\" (UniqueName: \"kubernetes.io/projected/2b74e7b8-7738-4fc8-9b08-27179c470990-kube-api-access-mqs48\") pod \"multus-additional-cni-plugins-tbgsj\" (UID: \"2b74e7b8-7738-4fc8-9b08-27179c470990\") " pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.147326 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.147289 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:17.156090 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.156051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mrpkj" Sep 17 18:29:17.165170 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.165134 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" Sep 17 18:29:17.169486 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.169439 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-75d5b" Sep 17 18:29:17.175302 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.175277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" Sep 17 18:29:17.181264 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.181243 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:29:17.181379 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.181358 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" Sep 17 18:29:17.188310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.188289 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vhft5" Sep 17 18:29:17.195478 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.195439 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r82qp" Sep 17 18:29:17.203185 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.203159 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:17.468111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.468031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:17.468238 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.468150 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:17.468238 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.468205 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret podName:8747e254-9790-4236-87a0-9955cb416dde nodeName:}" failed. No retries permitted until 2026-09-17 18:29:18.468192063 +0000 UTC m=+5.747843415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret") pod "global-pull-secret-syncer-6hvl9" (UID: "8747e254-9790-4236-87a0-9955cb416dde") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:17.568533 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.568493 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:17.568734 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.568564 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:17.568734 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.568702 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:17.568734 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.568722 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:17.568734 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.568730 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:17.568913 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.568746 2568 projected.go:196] Error preparing data for projected volume kube-api-access-mb4hp for pod openshift-network-diagnostics/network-check-target-wh8zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:17.568913 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.568790 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs podName:370e5c7b-6a22-41a2-bbe8-67e7f11bc373 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:18.568775168 +0000 UTC m=+5.848426534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs") pod "network-metrics-daemon-zq6bz" (UID: "370e5c7b-6a22-41a2-bbe8-67e7f11bc373") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:17.568913 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.568806 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp podName:00162636-5bc2-435a-8fbe-39f28f63641d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:18.568799784 +0000 UTC m=+5.848451136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb4hp" (UniqueName: "kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp") pod "network-check-target-wh8zt" (UID: "00162636-5bc2-435a-8fbe-39f28f63641d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:17.807347 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.807306 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b74e7b8_7738_4fc8_9b08_27179c470990.slice/crio-e931a91e3890b2b352cf083ea15261ca3bbd9433069a7c4c37b6e389f710d67f WatchSource:0}: Error finding container e931a91e3890b2b352cf083ea15261ca3bbd9433069a7c4c37b6e389f710d67f: Status 404 returned error can't find the container with id e931a91e3890b2b352cf083ea15261ca3bbd9433069a7c4c37b6e389f710d67f Sep 17 18:29:17.809027 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.809001 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a5d8fc_8768_4c7d_971a_485e2cd09de7.slice/crio-a8528ee53b67396741247dae4f68fa7b5ba03e20a569a7515180782c8dfe7d47 WatchSource:0}: Error finding container a8528ee53b67396741247dae4f68fa7b5ba03e20a569a7515180782c8dfe7d47: Status 404 returned error can't find the container with id a8528ee53b67396741247dae4f68fa7b5ba03e20a569a7515180782c8dfe7d47 Sep 17 18:29:17.810301 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.810275 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63869ab_2129_4c22_bdde_754ea60a7cf6.slice/crio-6271fb4416447865084b2156b8d8d97288924c261e280f9cc6037e4e3be9272c WatchSource:0}: Error finding container 6271fb4416447865084b2156b8d8d97288924c261e280f9cc6037e4e3be9272c: Status 404 returned error can't find the container with id 6271fb4416447865084b2156b8d8d97288924c261e280f9cc6037e4e3be9272c Sep 17 18:29:17.813010 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.812968 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092bdb7d_656a_4900_8af6_f0a03eab5aa9.slice/crio-1f1dfd1cacbcf9f905df2e93675a32381fdcb506a4430dc389f17148fb1726a9 WatchSource:0}: Error finding container 1f1dfd1cacbcf9f905df2e93675a32381fdcb506a4430dc389f17148fb1726a9: Status 404 returned error can't find the container with id 1f1dfd1cacbcf9f905df2e93675a32381fdcb506a4430dc389f17148fb1726a9 Sep 17 18:29:17.814116 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.813808 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d8596e_c089_4b0b_89ab_68002f3c78d8.slice/crio-6947da9d825cf0eb71d1d47f1b95b5d6f9e20b133a97d70d500e2c05f5f245d4 WatchSource:0}: Error finding container 6947da9d825cf0eb71d1d47f1b95b5d6f9e20b133a97d70d500e2c05f5f245d4: Status 404 returned error can't find the container with id 6947da9d825cf0eb71d1d47f1b95b5d6f9e20b133a97d70d500e2c05f5f245d4 Sep 17 18:29:17.814912 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.814882 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d928d4_4882_46ee_91c9_47e723e32b3a.slice/crio-ce01852b2c4a7d3c0c796aa90607006c3901bd12ed6839f0bb9ceb0072c3e0eb WatchSource:0}: Error finding container ce01852b2c4a7d3c0c796aa90607006c3901bd12ed6839f0bb9ceb0072c3e0eb: Status 404 returned error can't find the container with id ce01852b2c4a7d3c0c796aa90607006c3901bd12ed6839f0bb9ceb0072c3e0eb Sep 17 18:29:17.815898 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.815822 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ff0fa7_5e71_45db_a096_722ef629cfca.slice/crio-45ab17d7653e9287803e4cc112ca40ecfe21892ba3c153d6d22fcf73eea156cb WatchSource:0}: Error finding container 45ab17d7653e9287803e4cc112ca40ecfe21892ba3c153d6d22fcf73eea156cb: Status 404 returned error can't find the container with id 45ab17d7653e9287803e4cc112ca40ecfe21892ba3c153d6d22fcf73eea156cb Sep 17 18:29:17.816888 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.816867 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc672c626_1477_4611_a266_16e533c89d87.slice/crio-9c9505cb9c11fdb7b5ccd736c36db6e7a6f0ea4163f56783300fc2268f67eaed WatchSource:0}: Error finding container 9c9505cb9c11fdb7b5ccd736c36db6e7a6f0ea4163f56783300fc2268f67eaed: Status 404 returned error can't find the container with id 9c9505cb9c11fdb7b5ccd736c36db6e7a6f0ea4163f56783300fc2268f67eaed Sep 17 18:29:17.817710 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:17.817689 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ab30d6_ddf1_4b86_9c27_53625f671ccc.slice/crio-fb8a4239e6de57c11bb07590450ee59920c208163c1cbc735542a743eceac5cc WatchSource:0}: Error finding container fb8a4239e6de57c11bb07590450ee59920c208163c1cbc735542a743eceac5cc: Status 404 returned error can't find the container with id fb8a4239e6de57c11bb07590450ee59920c208163c1cbc735542a743eceac5cc Sep 17 18:29:17.887568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.887517 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:16 +0000 UTC" deadline="2028-06-19 05:08:44.60589464 +0000 UTC" Sep 17 18:29:17.887568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.887563 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15370h39m26.718334039s" Sep 17 18:29:17.979288 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.979073 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:17.979462 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.979130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:17.979923 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.979871 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:17.980015 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:17.979966 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:17.987805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.987775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-75d5b" event={"ID":"c672c626-1477-4611-a266-16e533c89d87","Type":"ContainerStarted","Data":"9c9505cb9c11fdb7b5ccd736c36db6e7a6f0ea4163f56783300fc2268f67eaed"} Sep 17 18:29:17.989150 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.989124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"ce01852b2c4a7d3c0c796aa90607006c3901bd12ed6839f0bb9ceb0072c3e0eb"} Sep 17 18:29:17.990752 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.990718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r82qp" event={"ID":"e3d8596e-c089-4b0b-89ab-68002f3c78d8","Type":"ContainerStarted","Data":"6947da9d825cf0eb71d1d47f1b95b5d6f9e20b133a97d70d500e2c05f5f245d4"} Sep 17 18:29:17.992070 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.992049 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbdzm" event={"ID":"d63869ab-2129-4c22-bdde-754ea60a7cf6","Type":"ContainerStarted","Data":"6271fb4416447865084b2156b8d8d97288924c261e280f9cc6037e4e3be9272c"} Sep 17 18:29:17.994100 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.994067 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" event={"ID":"1f316321b0ddd050eb379ecf12c03828","Type":"ContainerStarted","Data":"c4b7c65fad0aa3155d8ba00c9e92c132bd0c7ad442975996e6c13757190fe8ee"} Sep 17 18:29:17.995360 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.995329 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" event={"ID":"f8ab30d6-ddf1-4b86-9c27-53625f671ccc","Type":"ContainerStarted","Data":"fb8a4239e6de57c11bb07590450ee59920c208163c1cbc735542a743eceac5cc"} Sep 17 18:29:17.996650 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.996624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" event={"ID":"00ff0fa7-5e71-45db-a096-722ef629cfca","Type":"ContainerStarted","Data":"45ab17d7653e9287803e4cc112ca40ecfe21892ba3c153d6d22fcf73eea156cb"} Sep 17 18:29:17.997952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.997928 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vhft5" event={"ID":"092bdb7d-656a-4900-8af6-f0a03eab5aa9","Type":"ContainerStarted","Data":"1f1dfd1cacbcf9f905df2e93675a32381fdcb506a4430dc389f17148fb1726a9"} Sep 17 18:29:17.999310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:17.999289 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mrpkj" event={"ID":"01a5d8fc-8768-4c7d-971a-485e2cd09de7","Type":"ContainerStarted","Data":"a8528ee53b67396741247dae4f68fa7b5ba03e20a569a7515180782c8dfe7d47"} Sep 17 18:29:18.000407 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.000385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerStarted","Data":"e931a91e3890b2b352cf083ea15261ca3bbd9433069a7c4c37b6e389f710d67f"} Sep 17 18:29:18.007872 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.007833 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" podStartSLOduration=2.007821162 podStartE2EDuration="2.007821162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:18.007765471 +0000 UTC m=+5.287416844" watchObservedRunningTime="2026-09-17 18:29:18.007821162 +0000 UTC m=+5.287472536" Sep 17 18:29:18.476199 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.476110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:18.476377 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.476278 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:18.476377 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.476343 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret podName:8747e254-9790-4236-87a0-9955cb416dde nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.476324839 +0000 UTC m=+7.755976202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret") pod "global-pull-secret-syncer-6hvl9" (UID: "8747e254-9790-4236-87a0-9955cb416dde") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.576664 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.576764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.576936 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.576957 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.576971 2568 projected.go:196] Error preparing data for projected volume kube-api-access-mb4hp for pod openshift-network-diagnostics/network-check-target-wh8zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.577030 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp podName:00162636-5bc2-435a-8fbe-39f28f63641d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.577012698 +0000 UTC m=+7.856664050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb4hp" (UniqueName: "kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp") pod "network-check-target-wh8zt" (UID: "00162636-5bc2-435a-8fbe-39f28f63641d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.577437 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:18.577537 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.577504 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs podName:370e5c7b-6a22-41a2-bbe8-67e7f11bc373 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.577489165 +0000 UTC m=+7.857140522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs") pod "network-metrics-daemon-zq6bz" (UID: "370e5c7b-6a22-41a2-bbe8-67e7f11bc373") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:18.888722 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.888622 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:16 +0000 UTC" deadline="2028-05-11 09:01:58.531864953 +0000 UTC" Sep 17 18:29:18.888722 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.888666 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14438h32m39.643203165s" Sep 17 18:29:18.986751 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:18.985599 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:18.986751 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:18.985795 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:19.018863 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:19.018825 2568 generic.go:356] "Generic (PLEG): container finished" podID="4098de4361d529da9156a4482a8f5692" containerID="af8d15b8806d622952b6b42f9d4dbc5df5b259971db73322de4025548a8d910d" exitCode=0 Sep 17 18:29:19.019925 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:19.019894 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"4098de4361d529da9156a4482a8f5692","Type":"ContainerDied","Data":"af8d15b8806d622952b6b42f9d4dbc5df5b259971db73322de4025548a8d910d"} Sep 17 18:29:19.979458 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:19.979265 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:19.979959 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:19.979600 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:19.980097 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:19.980037 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:19.980158 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:19.980131 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:20.053525 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:20.052956 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"4098de4361d529da9156a4482a8f5692","Type":"ContainerStarted","Data":"dc5ee5edf919949d3d3e260e8c8501a7e51bff208eeeebc2f3cc9ef5c35e191d"} Sep 17 18:29:20.069906 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:20.069847 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" podStartSLOduration=4.069823521 podStartE2EDuration="4.069823521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:20.069047867 +0000 UTC m=+7.348699243" watchObservedRunningTime="2026-09-17 18:29:20.069823521 +0000 UTC m=+7.349474898" Sep 17 18:29:20.502342 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:20.502302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:20.502602 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.502471 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:20.502602 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.502538 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret podName:8747e254-9790-4236-87a0-9955cb416dde nodeName:}" failed. No retries permitted until 2026-09-17 18:29:24.502520602 +0000 UTC m=+11.782171960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret") pod "global-pull-secret-syncer-6hvl9" (UID: "8747e254-9790-4236-87a0-9955cb416dde") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:20.602855 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:20.602783 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:20.602855 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:20.602849 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:20.603096 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.602995 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:20.603096 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.603059 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs podName:370e5c7b-6a22-41a2-bbe8-67e7f11bc373 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:24.603039425 +0000 UTC m=+11.882690777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs") pod "network-metrics-daemon-zq6bz" (UID: "370e5c7b-6a22-41a2-bbe8-67e7f11bc373") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:20.603422 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.603299 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:20.603422 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.603330 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:20.603422 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.603344 2568 projected.go:196] Error preparing data for projected volume kube-api-access-mb4hp for pod openshift-network-diagnostics/network-check-target-wh8zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:20.603422 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.603399 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp podName:00162636-5bc2-435a-8fbe-39f28f63641d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:24.603379754 +0000 UTC m=+11.883031112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb4hp" (UniqueName: "kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp") pod "network-check-target-wh8zt" (UID: "00162636-5bc2-435a-8fbe-39f28f63641d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:20.978882 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:20.978800 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:20.979050 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:20.978948 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:21.979388 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:21.979297 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:21.979907 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:21.979468 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:21.979907 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:21.979862 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:21.980021 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:21.979966 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:22.979482 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:22.979429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:22.979947 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:22.979586 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:23.979392 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:23.979350 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:23.979560 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:23.979405 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:23.979560 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:23.979522 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:23.979983 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:23.979628 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:24.537827 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:24.537683 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:24.537827 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.537843 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:24.538088 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.537912 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret podName:8747e254-9790-4236-87a0-9955cb416dde nodeName:}" failed. No retries permitted until 2026-09-17 18:29:32.537895029 +0000 UTC m=+19.817546383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret") pod "global-pull-secret-syncer-6hvl9" (UID: "8747e254-9790-4236-87a0-9955cb416dde") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:24.639021 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:24.638983 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:24.639191 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:24.639057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:24.639191 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.639177 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:24.639326 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.639192 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:24.639326 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.639206 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:24.639326 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.639223 2568 projected.go:196] Error preparing data for projected volume kube-api-access-mb4hp for pod openshift-network-diagnostics/network-check-target-wh8zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:24.639326 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.639255 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs podName:370e5c7b-6a22-41a2-bbe8-67e7f11bc373 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:32.639236633 +0000 UTC m=+19.918888003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs") pod "network-metrics-daemon-zq6bz" (UID: "370e5c7b-6a22-41a2-bbe8-67e7f11bc373") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:24.639326 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.639279 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp podName:00162636-5bc2-435a-8fbe-39f28f63641d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:32.639262389 +0000 UTC m=+19.918913745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb4hp" (UniqueName: "kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp") pod "network-check-target-wh8zt" (UID: "00162636-5bc2-435a-8fbe-39f28f63641d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:24.980474 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:24.979914 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:24.980474 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:24.980049 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:25.979466 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:25.979392 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:25.979711 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:25.979544 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:25.979711 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:25.979389 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:25.979843 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:25.979727 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:26.978646 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:26.978610 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:26.979102 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:26.978737 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:27.979501 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:27.979402 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:27.979501 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:27.979470 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:27.979975 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:27.979555 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:27.979975 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:27.979692 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:28.978877 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:28.978842 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:28.979084 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:28.978986 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:29.978667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:29.978621 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:29.978667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:29.978645 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:29.979217 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:29.978804 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:29.979217 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:29.978930 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:30.979135 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:30.979098 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:30.979626 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:30.979251 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:31.978879 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:31.978842 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:31.979052 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:31.978852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:31.979052 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:31.979002 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:31.979052 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:31.979035 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:32.602886 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:32.602843 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:32.603068 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.603001 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:32.603125 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.603084 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret podName:8747e254-9790-4236-87a0-9955cb416dde nodeName:}" failed. No retries permitted until 2026-09-17 18:29:48.603061945 +0000 UTC m=+35.882713313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret") pod "global-pull-secret-syncer-6hvl9" (UID: "8747e254-9790-4236-87a0-9955cb416dde") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:32.703234 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:32.703193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:32.703418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:32.703264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:32.703418 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.703376 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:32.703418 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.703399 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:32.703598 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.703403 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:32.703598 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.703493 2568 projected.go:196] Error preparing data for projected volume kube-api-access-mb4hp for pod openshift-network-diagnostics/network-check-target-wh8zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:32.703598 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.703531 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs podName:370e5c7b-6a22-41a2-bbe8-67e7f11bc373 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:48.703510735 +0000 UTC m=+35.983162100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs") pod "network-metrics-daemon-zq6bz" (UID: "370e5c7b-6a22-41a2-bbe8-67e7f11bc373") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:32.703598 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.703554 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp podName:00162636-5bc2-435a-8fbe-39f28f63641d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:48.703542918 +0000 UTC m=+35.983194277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb4hp" (UniqueName: "kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp") pod "network-check-target-wh8zt" (UID: "00162636-5bc2-435a-8fbe-39f28f63641d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:32.982848 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:32.982761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:32.983305 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:32.982900 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:33.978810 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:33.978773 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:33.978997 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:33.978776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:33.978997 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:33.978905 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:33.978997 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:33.978987 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:34.981578 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:34.981552 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:34.981928 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:34.981647 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:35.979548 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:35.979212 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:35.979702 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:35.979257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:35.979702 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:35.979655 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:35.979779 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:35.979710 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:36.085038 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.084942 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-75d5b" event={"ID":"c672c626-1477-4611-a266-16e533c89d87","Type":"ContainerStarted","Data":"613c8596a59be0460be7d5d83c61ba58e2c89190394a133ad94d8702de9da654"} Sep 17 18:29:36.088317 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088294 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:29:36.088698 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088675 2568 generic.go:356] "Generic (PLEG): container finished" podID="52d928d4-4882-46ee-91c9-47e723e32b3a" containerID="5e13362cb5c5027683bbce0c7fefe562f8957b2e6facb3585a0c3739b61b8a36" exitCode=1 Sep 17 18:29:36.088790 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"3ebde4affeaa6a1675803e138dba1110bd008062abfdae4f93596ea08f35bf18"} Sep 17 18:29:36.088838 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088808 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"13dee66cc321840478595708af0b15ad7d680442353899c79424bc0de3868daa"} Sep 17 18:29:36.088897 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"b83fd1098456f9f27ea00388c6f3802241c3aba7c76f4ee641bf5b41c7a2316b"} Sep 17 18:29:36.088941 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088905 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"15428c30624bea27401cd98cdc64c92881c1c9529fd70d0193522883a7b36cad"} Sep 17 18:29:36.088941 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088937 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerDied","Data":"5e13362cb5c5027683bbce0c7fefe562f8957b2e6facb3585a0c3739b61b8a36"} Sep 17 18:29:36.089042 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.088975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"7b5f813f9f5d9faa4e564694118b684934b2ef0f34c2b5fbabd981f2f5ec5452"} Sep 17 18:29:36.090367 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.090337 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r82qp" event={"ID":"e3d8596e-c089-4b0b-89ab-68002f3c78d8","Type":"ContainerStarted","Data":"989af08efffe6c05ed92fc3a7bcfc7cdf9aefd91509443fc8e5e4949521bb49e"} Sep 17 18:29:36.091915 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.091891 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbdzm" event={"ID":"d63869ab-2129-4c22-bdde-754ea60a7cf6","Type":"ContainerStarted","Data":"83d993430d100758abe0fc83ace67e7a2d63e161541b53f8473daa415dc1c731"} Sep 17 18:29:36.093491 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.093468 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" event={"ID":"f8ab30d6-ddf1-4b86-9c27-53625f671ccc","Type":"ContainerStarted","Data":"db5a854de927bb2803091427f5620391988fd4751f1ccf49b028d48aca426353"} Sep 17 18:29:36.094872 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.094848 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" event={"ID":"00ff0fa7-5e71-45db-a096-722ef629cfca","Type":"ContainerStarted","Data":"9502aa67ccde5f336d6c51381e21c2ff635483738b4cf2847adc82920c340be5"} Sep 17 18:29:36.096493 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.096470 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vhft5" event={"ID":"092bdb7d-656a-4900-8af6-f0a03eab5aa9","Type":"ContainerStarted","Data":"94268e95c09fca7d1772d21d4d1d08f3b182caa614c1077ced3c4b388fed3b80"} Sep 17 18:29:36.099933 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.099900 2568 generic.go:356] "Generic (PLEG): container finished" podID="2b74e7b8-7738-4fc8-9b08-27179c470990" containerID="9083b44a991247b21738768d49b802aaec53a41c2d575cc74809bb5626b2cca4" exitCode=0 Sep 17 18:29:36.100063 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.099944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerDied","Data":"9083b44a991247b21738768d49b802aaec53a41c2d575cc74809bb5626b2cca4"} Sep 17 18:29:36.101587 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.101541 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-75d5b" podStartSLOduration=3.853024356 podStartE2EDuration="21.101523131s" totalImagesPullingTime="17.248498775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:36.100231847 +0000 UTC m=+23.379883223" watchObservedRunningTime="2026-09-17 18:29:36.101523131 +0000 UTC m=+23.381174510" Sep 17 18:29:36.120587 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.120541 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mvqf7" podStartSLOduration=3.872634782 podStartE2EDuration="21.120526223s" totalImagesPullingTime="17.247891441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:36.12015367 +0000 UTC m=+23.399805045" watchObservedRunningTime="2026-09-17 18:29:36.120526223 +0000 UTC m=+23.400177601" Sep 17 18:29:36.139856 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.139805 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vhft5" podStartSLOduration=3.888075314 podStartE2EDuration="21.139786143s" totalImagesPullingTime="17.251710829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:36.138765131 +0000 UTC m=+23.418416507" watchObservedRunningTime="2026-09-17 18:29:36.139786143 +0000 UTC m=+23.419437519" Sep 17 18:29:36.176379 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.176333 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r82qp" podStartSLOduration=3.879107453 podStartE2EDuration="21.176316707s" totalImagesPullingTime="17.297209254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:36.175742477 +0000 UTC m=+23.455393852" watchObservedRunningTime="2026-09-17 18:29:36.176316707 +0000 UTC m=+23.455968080" Sep 17 18:29:36.238293 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.238245 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sbdzm" podStartSLOduration=3.983346659 podStartE2EDuration="21.238230254s" totalImagesPullingTime="17.254883595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:36.211093302 +0000 UTC m=+23.490744687" watchObservedRunningTime="2026-09-17 18:29:36.238230254 +0000 UTC m=+23.517881689" Sep 17 18:29:36.579166 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.579131 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 18:29:36.945688 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.945579 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:36.579159105Z","UUID":"d5a0bed5-bf96-4079-9a62-452f1e0e9089","Handler":null,"Name":"","Endpoint":""} Sep 17 18:29:36.947678 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.947656 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 18:29:36.947824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.947689 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 18:29:36.983254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:36.983228 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:36.983417 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:36.983347 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:37.104072 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:37.104041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mrpkj" event={"ID":"01a5d8fc-8768-4c7d-971a-485e2cd09de7","Type":"ContainerStarted","Data":"543f119a7db3baccb51a51aa7e051fccda6c7100e99de29f3eba989d7326f3c6"} Sep 17 18:29:37.106209 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:37.106179 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" event={"ID":"f8ab30d6-ddf1-4b86-9c27-53625f671ccc","Type":"ContainerStarted","Data":"7ac5a7f08ec870ced924360948c8981ef733ef020ad7416d2cb66b27e797dcd9"} Sep 17 18:29:37.119882 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:37.119840 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mrpkj" podStartSLOduration=4.8639013989999995 podStartE2EDuration="22.119821008s" totalImagesPullingTime="17.255919609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:37.119212174 +0000 UTC m=+24.398863549" watchObservedRunningTime="2026-09-17 18:29:37.119821008 +0000 UTC m=+24.399472383" Sep 17 18:29:37.298566 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:37.298481 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:37.299111 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:37.299077 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:37.979540 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:37.979464 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:37.979540 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:37.979509 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:37.979770 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:37.979596 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:37.979770 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:37.979723 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:38.111816 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:38.111757 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" event={"ID":"f8ab30d6-ddf1-4b86-9c27-53625f671ccc","Type":"ContainerStarted","Data":"a7f0557a8a7d80989b2da352620cfd7a38110cbe2e8bec4c579be4445940f6b0"} Sep 17 18:29:38.116018 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:38.115997 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:29:38.116706 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:38.116681 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:38.117144 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:38.117127 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sbdzm" Sep 17 18:29:38.131001 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:38.130951 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmn6f" podStartSLOduration=4.047928106 podStartE2EDuration="23.130935367s" totalImagesPullingTime="19.083007261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:38.130769138 +0000 UTC m=+25.410420517" watchObservedRunningTime="2026-09-17 18:29:38.130935367 +0000 UTC m=+25.410586742" Sep 17 18:29:38.979062 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:38.979028 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:38.979227 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:38.979193 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:39.121651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:39.121624 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:29:39.122080 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:39.122035 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"4d3cdb813e84be583ebe878256b6a95c8d672c49af3285d55d61ff4164a61399"} Sep 17 18:29:39.978934 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:39.978704 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:39.979094 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:39.978733 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:39.979094 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:39.979059 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:39.979178 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:39.979109 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:40.978590 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:40.978525 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:40.979223 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:40.978641 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:41.130022 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.129992 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:29:41.130382 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.130359 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"ded32aa445e9320412b944b8571df6d700639af78d641d905da2043d5e305bb3"} Sep 17 18:29:41.130639 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.130621 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:41.130718 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.130689 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:41.130804 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.130787 2568 scope.go:117] "RemoveContainer" containerID="5e13362cb5c5027683bbce0c7fefe562f8957b2e6facb3585a0c3739b61b8a36" Sep 17 18:29:41.132353 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.132330 2568 generic.go:356] "Generic (PLEG): container finished" podID="2b74e7b8-7738-4fc8-9b08-27179c470990" containerID="9acc9f2ab8eea835bf3d7b0c0a7fe1954ed716937b1b3aff38e5ae69b464e559" exitCode=0 Sep 17 18:29:41.132435 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.132362 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerDied","Data":"9acc9f2ab8eea835bf3d7b0c0a7fe1954ed716937b1b3aff38e5ae69b464e559"} Sep 17 18:29:41.147816 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.147793 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:41.978620 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.978425 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:41.979019 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:41.978463 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:41.979019 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:41.978722 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:41.979019 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:41.978756 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:42.138556 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.138490 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:29:42.138892 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.138865 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" event={"ID":"52d928d4-4882-46ee-91c9-47e723e32b3a","Type":"ContainerStarted","Data":"8e9a727b484aa236aefd828c66df5da08c9724c15dc9b33ee31fbd59527d5d15"} Sep 17 18:29:42.139182 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.139165 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:42.141191 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.141168 2568 generic.go:356] "Generic (PLEG): container finished" podID="2b74e7b8-7738-4fc8-9b08-27179c470990" containerID="fb3c7d5cb0fff1c9e2884ec5cc4aeb0ce7b4329a1a594f338008545605088de4" exitCode=0 Sep 17 18:29:42.141298 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.141203 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerDied","Data":"fb3c7d5cb0fff1c9e2884ec5cc4aeb0ce7b4329a1a594f338008545605088de4"} Sep 17 18:29:42.154820 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.154800 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:29:42.166905 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.166865 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" podStartSLOduration=9.596014384 podStartE2EDuration="27.166850509s" totalImagesPullingTime="17.570836125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.165671986 +0000 UTC m=+29.445323360" watchObservedRunningTime="2026-09-17 18:29:42.166850509 +0000 UTC m=+29.446501916" Sep 17 18:29:42.677695 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.677669 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wh8zt"] Sep 17 18:29:42.677808 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.677780 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:42.677901 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:42.677882 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:42.680817 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.680785 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zq6bz"] Sep 17 18:29:42.680944 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.680901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:42.681072 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:42.681049 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:42.690884 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.690857 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hvl9"] Sep 17 18:29:42.691004 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:42.690977 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:42.691081 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:42.691061 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:43.146094 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:43.146007 2568 generic.go:356] "Generic (PLEG): container finished" podID="2b74e7b8-7738-4fc8-9b08-27179c470990" containerID="01525177c3db5bb1ab2ae1c2c8859b2623661df6c778a14ca3bb4db8b67df38e" exitCode=0 Sep 17 18:29:43.146466 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:43.146091 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerDied","Data":"01525177c3db5bb1ab2ae1c2c8859b2623661df6c778a14ca3bb4db8b67df38e"} Sep 17 18:29:43.979019 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:43.978973 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:43.979019 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:43.979023 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:43.979286 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:43.979128 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:43.979286 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:43.979247 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:44.979345 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:44.979310 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:44.979814 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:44.979417 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:45.979295 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:45.979247 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:45.979533 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:45.979256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:45.979533 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:45.979407 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:45.979956 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:45.979524 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:46.979184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:46.979144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:46.979487 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:46.979273 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wh8zt" podUID="00162636-5bc2-435a-8fbe-39f28f63641d" Sep 17 18:29:47.978929 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:47.978738 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:47.979320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:47.978781 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:47.979320 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:47.979015 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zq6bz" podUID="370e5c7b-6a22-41a2-bbe8-67e7f11bc373" Sep 17 18:29:47.979320 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:47.979070 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hvl9" podUID="8747e254-9790-4236-87a0-9955cb416dde" Sep 17 18:29:48.131775 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.131695 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeReady" Sep 17 18:29:48.131956 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.131801 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 18:29:48.162030 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.161998 2568 generic.go:356] "Generic (PLEG): container finished" podID="2b74e7b8-7738-4fc8-9b08-27179c470990" containerID="a59b0872a0ee185cd5376f312b2009e2661e98234de8f5496acdc6684e993976" exitCode=0 Sep 17 18:29:48.162204 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.162056 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerDied","Data":"a59b0872a0ee185cd5376f312b2009e2661e98234de8f5496acdc6684e993976"} Sep 17 18:29:48.166818 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.166793 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68bb7dd5cf-k5s9b"] Sep 17 18:29:48.168248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.168233 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.170568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.170533 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 18:29:48.170568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.170544 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fsz7t\"" Sep 17 18:29:48.170568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.170547 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 18:29:48.170765 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.170567 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 18:29:48.175425 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.175406 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 18:29:48.181328 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.181304 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rnxzj"] Sep 17 18:29:48.189170 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.189139 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68bb7dd5cf-k5s9b"] Sep 17 18:29:48.189301 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.189259 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.192285 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.192260 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 18:29:48.192285 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.192274 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qrnqw\"" Sep 17 18:29:48.192633 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.192346 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 18:29:48.192633 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.192578 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 18:29:48.198613 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.198589 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rnxzj"] Sep 17 18:29:48.277735 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.277710 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbkr4"] Sep 17 18:29:48.281167 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.281150 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.283642 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.283620 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 18:29:48.283844 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.283828 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 18:29:48.283984 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.283971 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f4hc6\"" Sep 17 18:29:48.290016 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.289994 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbkr4"] Sep 17 18:29:48.322424 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322388 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-image-registry-private-configuration\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.322611 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b748973-196a-4cf2-a06f-67b2588fc4ef-cert\") pod \"ingress-canary-rnxzj\" (UID: \"5b748973-196a-4cf2-a06f-67b2588fc4ef\") " pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.322611 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-certificates\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.322611 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-installation-pull-secrets\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.322876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnpb\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-kube-api-access-ggnpb\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.322876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prw85\" (UniqueName: \"kubernetes.io/projected/5b748973-196a-4cf2-a06f-67b2588fc4ef-kube-api-access-prw85\") pod \"ingress-canary-rnxzj\" (UID: \"5b748973-196a-4cf2-a06f-67b2588fc4ef\") " pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.322876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.322876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322774 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-trusted-ca\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.322876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322801 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-bound-sa-token\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.322876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.322831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-ca-trust-extracted\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.394108 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.394028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbdzm_d63869ab-2129-4c22-bdde-754ea60a7cf6/konnectivity-agent/0.log" limit=1024 Sep 17 18:29:48.424068 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-tmp-dir\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.424175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-image-registry-private-configuration\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b748973-196a-4cf2-a06f-67b2588fc4ef-cert\") pod \"ingress-canary-rnxzj\" (UID: \"5b748973-196a-4cf2-a06f-67b2588fc4ef\") " pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.424241 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-certificates\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424241 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-installation-pull-secrets\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424241 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424220 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnpb\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-kube-api-access-ggnpb\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424382 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-config-volume\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.424382 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfstl\" (UniqueName: \"kubernetes.io/projected/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-kube-api-access-bfstl\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.424520 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prw85\" (UniqueName: \"kubernetes.io/projected/5b748973-196a-4cf2-a06f-67b2588fc4ef-kube-api-access-prw85\") pod \"ingress-canary-rnxzj\" (UID: \"5b748973-196a-4cf2-a06f-67b2588fc4ef\") " pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.424520 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424520 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-trusted-ca\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424520 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-bound-sa-token\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424520 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424514 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.424753 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424542 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-ca-trust-extracted\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.424753 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.424701 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:48.424753 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.424716 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68bb7dd5cf-k5s9b: secret "image-registry-tls" not found Sep 17 18:29:48.424883 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.424768 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls podName:d22ec93c-7ad1-4ed4-880b-e60c3faca6d3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:48.92474953 +0000 UTC m=+36.204400905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls") pod "image-registry-68bb7dd5cf-k5s9b" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3") : secret "image-registry-tls" not found Sep 17 18:29:48.424883 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.424862 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-certificates\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.425300 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.425277 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-ca-trust-extracted\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.425848 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.425832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-trusted-ca\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.428291 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.428259 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-image-registry-private-configuration\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.428379 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.428301 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b748973-196a-4cf2-a06f-67b2588fc4ef-cert\") pod \"ingress-canary-rnxzj\" (UID: \"5b748973-196a-4cf2-a06f-67b2588fc4ef\") " pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.428379 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.428330 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-installation-pull-secrets\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.434536 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.434517 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prw85\" (UniqueName: \"kubernetes.io/projected/5b748973-196a-4cf2-a06f-67b2588fc4ef-kube-api-access-prw85\") pod \"ingress-canary-rnxzj\" (UID: \"5b748973-196a-4cf2-a06f-67b2588fc4ef\") " pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.434966 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.434946 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnpb\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-kube-api-access-ggnpb\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.435033 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.434977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-bound-sa-token\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.489571 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.489540 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbdzm_d63869ab-2129-4c22-bdde-754ea60a7cf6/konnectivity-agent/0.log" limit=1024 Sep 17 18:29:48.501315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.501291 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rnxzj" Sep 17 18:29:48.524904 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.524880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.525043 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.524914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-tmp-dir\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.525043 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.524955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-config-volume\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.525043 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.524991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfstl\" (UniqueName: \"kubernetes.io/projected/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-kube-api-access-bfstl\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.525207 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.525048 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:48.525207 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.525127 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls podName:1cc0cf8d-03e2-489e-9a01-cb24fa4ee382 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:49.025105451 +0000 UTC m=+36.304756826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls") pod "dns-default-dbkr4" (UID: "1cc0cf8d-03e2-489e-9a01-cb24fa4ee382") : secret "dns-default-metrics-tls" not found Sep 17 18:29:48.525350 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.525331 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-tmp-dir\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.525529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.525512 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-config-volume\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.533318 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.533293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfstl\" (UniqueName: \"kubernetes.io/projected/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-kube-api-access-bfstl\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:48.626079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.626052 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:48.626248 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.626177 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:48.626248 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.626225 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret podName:8747e254-9790-4236-87a0-9955cb416dde nodeName:}" failed. No retries permitted until 2026-09-17 18:30:20.626214097 +0000 UTC m=+67.905865462 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret") pod "global-pull-secret-syncer-6hvl9" (UID: "8747e254-9790-4236-87a0-9955cb416dde") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:29:48.727345 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.727247 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:48.727345 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.727324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:48.727523 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.727417 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:48.727523 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.727417 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:29:48.727523 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.727519 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:29:48.727620 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.727531 2568 projected.go:196] Error preparing data for projected volume kube-api-access-mb4hp for pod openshift-network-diagnostics/network-check-target-wh8zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:48.727620 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.727501 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs podName:370e5c7b-6a22-41a2-bbe8-67e7f11bc373 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:20.727486447 +0000 UTC m=+68.007137812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs") pod "network-metrics-daemon-zq6bz" (UID: "370e5c7b-6a22-41a2-bbe8-67e7f11bc373") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:29:48.727620 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.727578 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp podName:00162636-5bc2-435a-8fbe-39f28f63641d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:20.727568704 +0000 UTC m=+68.007220056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb4hp" (UniqueName: "kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp") pod "network-check-target-wh8zt" (UID: "00162636-5bc2-435a-8fbe-39f28f63641d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:29:48.776766 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.776735 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rnxzj"] Sep 17 18:29:48.780927 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:48.780888 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b748973_196a_4cf2_a06f_67b2588fc4ef.slice/crio-6349d28a2b6baffd87b901338a720b3b56e587d17ac2862513e596ea4f4ed639 WatchSource:0}: Error finding container 6349d28a2b6baffd87b901338a720b3b56e587d17ac2862513e596ea4f4ed639: Status 404 returned error can't find the container with id 6349d28a2b6baffd87b901338a720b3b56e587d17ac2862513e596ea4f4ed639 Sep 17 18:29:48.929333 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.929293 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:48.929529 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.929426 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:48.929529 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.929438 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68bb7dd5cf-k5s9b: secret "image-registry-tls" not found Sep 17 18:29:48.929529 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:48.929514 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls podName:d22ec93c-7ad1-4ed4-880b-e60c3faca6d3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:49.929501211 +0000 UTC m=+37.209152579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls") pod "image-registry-68bb7dd5cf-k5s9b" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3") : secret "image-registry-tls" not found Sep 17 18:29:48.978765 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.978683 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:29:48.981349 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.981324 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhngp\"" Sep 17 18:29:48.981349 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.981339 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 18:29:48.982084 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:48.981343 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 18:29:49.029921 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.029895 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:49.030075 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:49.030050 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:49.030121 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:49.030114 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls podName:1cc0cf8d-03e2-489e-9a01-cb24fa4ee382 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:50.030097089 +0000 UTC m=+37.309748446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls") pod "dns-default-dbkr4" (UID: "1cc0cf8d-03e2-489e-9a01-cb24fa4ee382") : secret "dns-default-metrics-tls" not found Sep 17 18:29:49.168203 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.168169 2568 generic.go:356] "Generic (PLEG): container finished" podID="2b74e7b8-7738-4fc8-9b08-27179c470990" containerID="0365ca26f84fbbf3242f6bd95084b72fa4afc8ea872ef3579ec4e0c296108c0c" exitCode=0 Sep 17 18:29:49.168405 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.168263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerDied","Data":"0365ca26f84fbbf3242f6bd95084b72fa4afc8ea872ef3579ec4e0c296108c0c"} Sep 17 18:29:49.169898 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.169869 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rnxzj" event={"ID":"5b748973-196a-4cf2-a06f-67b2588fc4ef","Type":"ContainerStarted","Data":"6349d28a2b6baffd87b901338a720b3b56e587d17ac2862513e596ea4f4ed639"} Sep 17 18:29:49.937962 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.937763 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:49.938162 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:49.937936 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:49.938162 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:49.938034 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68bb7dd5cf-k5s9b: secret "image-registry-tls" not found Sep 17 18:29:49.938162 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:49.938106 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls podName:d22ec93c-7ad1-4ed4-880b-e60c3faca6d3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:51.93808418 +0000 UTC m=+39.217735545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls") pod "image-registry-68bb7dd5cf-k5s9b" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3") : secret "image-registry-tls" not found Sep 17 18:29:49.978780 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.978737 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:29:49.978950 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.978794 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:29:49.981326 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.981304 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 18:29:49.981468 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.981354 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 18:29:49.981468 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:49.981304 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmb9h\"" Sep 17 18:29:50.039264 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:50.039230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:50.039455 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:50.039392 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:50.039514 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:50.039487 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls podName:1cc0cf8d-03e2-489e-9a01-cb24fa4ee382 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:52.039462863 +0000 UTC m=+39.319114235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls") pod "dns-default-dbkr4" (UID: "1cc0cf8d-03e2-489e-9a01-cb24fa4ee382") : secret "dns-default-metrics-tls" not found Sep 17 18:29:50.176341 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:50.176303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" event={"ID":"2b74e7b8-7738-4fc8-9b08-27179c470990","Type":"ContainerStarted","Data":"bb427de6497d55ee7b4f158d7a2c0026d911c44e2481cb7d645ca30a86251486"} Sep 17 18:29:50.200134 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:50.200027 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tbgsj" podStartSLOduration=7.7902742 podStartE2EDuration="35.200004815s" totalImagesPullingTime="27.409730615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:50.198994017 +0000 UTC m=+37.478645394" watchObservedRunningTime="2026-09-17 18:29:50.200004815 +0000 UTC m=+37.479656190" Sep 17 18:29:50.322588 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:50.322552 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbdzm_d63869ab-2129-4c22-bdde-754ea60a7cf6/konnectivity-agent/0.log" limit=1024 Sep 17 18:29:51.180627 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:51.180594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rnxzj" event={"ID":"5b748973-196a-4cf2-a06f-67b2588fc4ef","Type":"ContainerStarted","Data":"a4a4728c7261f57260b625c4313aa4b6763da5f7010bee06ab0708e5003959c8"} Sep 17 18:29:51.198188 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:51.198141 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rnxzj" podStartSLOduration=1.118480221 podStartE2EDuration="3.198125754s" totalImagesPullingTime="2.079645533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:51.197083107 +0000 UTC m=+38.476734481" watchObservedRunningTime="2026-09-17 18:29:51.198125754 +0000 UTC m=+38.477777118" Sep 17 18:29:51.952115 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:51.952081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:51.952314 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:51.952249 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:51.952314 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:51.952282 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68bb7dd5cf-k5s9b: secret "image-registry-tls" not found Sep 17 18:29:51.952422 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:51.952353 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls podName:d22ec93c-7ad1-4ed4-880b-e60c3faca6d3 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:55.952330683 +0000 UTC m=+43.231982050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls") pod "image-registry-68bb7dd5cf-k5s9b" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3") : secret "image-registry-tls" not found Sep 17 18:29:52.053091 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:52.053059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:52.053252 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:52.053214 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:52.053333 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:52.053279 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls podName:1cc0cf8d-03e2-489e-9a01-cb24fa4ee382 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:56.053260787 +0000 UTC m=+43.332912154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls") pod "dns-default-dbkr4" (UID: "1cc0cf8d-03e2-489e-9a01-cb24fa4ee382") : secret "dns-default-metrics-tls" not found Sep 17 18:29:55.983229 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:55.983189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:29:55.983736 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:55.983316 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:29:55.983736 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:55.983327 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68bb7dd5cf-k5s9b: secret "image-registry-tls" not found Sep 17 18:29:55.983736 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:55.983374 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls podName:d22ec93c-7ad1-4ed4-880b-e60c3faca6d3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.983361592 +0000 UTC m=+51.263012943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls") pod "image-registry-68bb7dd5cf-k5s9b" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3") : secret "image-registry-tls" not found Sep 17 18:29:56.083895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:56.083869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:29:56.084002 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:56.083987 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:29:56.084053 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:56.084038 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls podName:1cc0cf8d-03e2-489e-9a01-cb24fa4ee382 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:04.084025147 +0000 UTC m=+51.363676513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls") pod "dns-default-dbkr4" (UID: "1cc0cf8d-03e2-489e-9a01-cb24fa4ee382") : secret "dns-default-metrics-tls" not found Sep 17 18:29:57.752424 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.752389 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf"] Sep 17 18:29:57.789700 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.789671 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-x9qpw"] Sep 17 18:29:57.789868 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.789798 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" Sep 17 18:29:57.792381 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.792354 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:57.792381 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.792378 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ss9gz\"" Sep 17 18:29:57.792614 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.792381 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:57.805095 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.805068 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf"] Sep 17 18:29:57.805095 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.805098 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-x9qpw"] Sep 17 18:29:57.805233 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.805170 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.807479 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.807341 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 18:29:57.807479 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.807364 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 18:29:57.807479 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.807394 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rwr7h\"" Sep 17 18:29:57.807479 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.807411 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 18:29:57.807866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.807852 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 18:29:57.811808 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.811786 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 18:29:57.837245 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.837215 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49"] Sep 17 18:29:57.858010 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.857982 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm"] Sep 17 18:29:57.858151 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.858106 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" Sep 17 18:29:57.868161 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.868141 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9vml8\"" Sep 17 18:29:57.879846 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.879826 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg"] Sep 17 18:29:57.879990 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.879972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:57.882800 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.882781 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 17 18:29:57.882896 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.882810 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:57.883182 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.883167 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 17 18:29:57.884029 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.884016 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2ssrd\"" Sep 17 18:29:57.887544 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.887527 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:57.896169 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.896142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x9z\" (UniqueName: \"kubernetes.io/projected/7869cb96-52c3-46c3-b799-1412fbce679b-kube-api-access-b8x9z\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.896261 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.896181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcbm\" (UniqueName: \"kubernetes.io/projected/68273c1d-c8b7-4149-a872-0ca8a179333d-kube-api-access-dlcbm\") pod \"volume-data-source-validator-5c4db8bdf9-b8vdf\" (UID: \"68273c1d-c8b7-4149-a872-0ca8a179333d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" Sep 17 18:29:57.896311 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.896288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7869cb96-52c3-46c3-b799-1412fbce679b-tmp\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.896425 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.896314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7869cb96-52c3-46c3-b799-1412fbce679b-snapshots\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.896425 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.896342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7869cb96-52c3-46c3-b799-1412fbce679b-serving-cert\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.896425 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.896395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7869cb96-52c3-46c3-b799-1412fbce679b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.896593 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.896439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7869cb96-52c3-46c3-b799-1412fbce679b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.903863 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.903842 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-695f977464-jzv52"] Sep 17 18:29:57.903963 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.903922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:57.906850 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.906830 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:57.906954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.906835 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 18:29:57.907798 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.907781 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:57.910621 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.910604 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kdm48\"" Sep 17 18:29:57.910785 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.910663 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49"] Sep 17 18:29:57.910785 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.910692 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg"] Sep 17 18:29:57.910785 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.910711 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm"] Sep 17 18:29:57.910785 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.910730 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-695f977464-jzv52"] Sep 17 18:29:57.910950 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.910786 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:57.924941 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.924921 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 18:29:57.925018 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.924989 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 18:29:57.925061 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.925041 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 18:29:57.925106 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.925060 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 18:29:57.925149 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.925064 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 18:29:57.925432 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.925415 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 18:29:57.925518 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.925505 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 18:29:57.925749 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.925734 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tb84v\"" Sep 17 18:29:57.955419 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.955383 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd"] Sep 17 18:29:57.974118 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.973889 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72"] Sep 17 18:29:57.974298 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.974245 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:57.976894 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.976843 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-79kvt\"" Sep 17 18:29:57.977936 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.977729 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 17 18:29:57.978055 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.977964 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 17 18:29:57.994831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.994807 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8"] Sep 17 18:29:57.994986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.994911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:57.997137 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x9z\" (UniqueName: \"kubernetes.io/projected/7869cb96-52c3-46c3-b799-1412fbce679b-kube-api-access-b8x9z\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.997280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcbm\" (UniqueName: \"kubernetes.io/projected/68273c1d-c8b7-4149-a872-0ca8a179333d-kube-api-access-dlcbm\") pod \"volume-data-source-validator-5c4db8bdf9-b8vdf\" (UID: \"68273c1d-c8b7-4149-a872-0ca8a179333d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" Sep 17 18:29:57.997280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1cff07-b71c-4c36-b79d-5376eb90acdd-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:57.997280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997190 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x687x\" (UniqueName: \"kubernetes.io/projected/fa1cff07-b71c-4c36-b79d-5376eb90acdd-kube-api-access-x687x\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:57.997280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997265 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-serving-cert\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:57.997529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997297 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:57.997529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997343 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 18:29:57.997529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997380 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tb45p\"" Sep 17 18:29:57.997529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7869cb96-52c3-46c3-b799-1412fbce679b-tmp\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.997529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7869cb96-52c3-46c3-b799-1412fbce679b-snapshots\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.997529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997470 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:57.997529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7869cb96-52c3-46c3-b799-1412fbce679b-serving-cert\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.997895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7869cb96-52c3-46c3-b799-1412fbce679b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.997895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgk66\" (UniqueName: \"kubernetes.io/projected/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-kube-api-access-vgk66\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:57.997895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6q29\" (UniqueName: \"kubernetes.io/projected/e1782d59-4fd9-4076-b0f4-54067c4cfd4c-kube-api-access-x6q29\") pod \"network-check-source-7cbc45fddc-6fv49\" (UID: \"e1782d59-4fd9-4076-b0f4-54067c4cfd4c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" Sep 17 18:29:57.997895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa1cff07-b71c-4c36-b79d-5376eb90acdd-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:57.997895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7869cb96-52c3-46c3-b799-1412fbce679b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.997895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7869cb96-52c3-46c3-b799-1412fbce679b-tmp\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.997895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.997768 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-config\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:57.998203 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.998031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7869cb96-52c3-46c3-b799-1412fbce679b-snapshots\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.998248 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.998225 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7869cb96-52c3-46c3-b799-1412fbce679b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:57.998431 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:57.998412 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7869cb96-52c3-46c3-b799-1412fbce679b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:58.001110 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.001091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7869cb96-52c3-46c3-b799-1412fbce679b-serving-cert\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:58.006013 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.005961 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x9z\" (UniqueName: \"kubernetes.io/projected/7869cb96-52c3-46c3-b799-1412fbce679b-kube-api-access-b8x9z\") pod \"insights-operator-6f9c644dd9-x9qpw\" (UID: \"7869cb96-52c3-46c3-b799-1412fbce679b\") " pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:58.006116 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.006090 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcbm\" (UniqueName: \"kubernetes.io/projected/68273c1d-c8b7-4149-a872-0ca8a179333d-kube-api-access-dlcbm\") pod \"volume-data-source-validator-5c4db8bdf9-b8vdf\" (UID: \"68273c1d-c8b7-4149-a872-0ca8a179333d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" Sep 17 18:29:58.015162 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.015133 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-ssmlh"] Sep 17 18:29:58.015285 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.015253 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.018204 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.018173 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 17 18:29:58.018330 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.018226 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nj488\"" Sep 17 18:29:58.018330 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.018259 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 18:29:58.018517 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.018494 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 18:29:58.019263 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.019248 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 17 18:29:58.034896 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.034873 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd"] Sep 17 18:29:58.034896 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.034899 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72"] Sep 17 18:29:58.035018 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.034918 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8"] Sep 17 18:29:58.035018 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.034932 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-ssmlh"] Sep 17 18:29:58.035018 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.034978 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.038127 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.038080 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 18:29:58.038127 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.038111 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 18:29:58.038334 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.038133 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 18:29:58.038334 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.038113 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:29:58.038334 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.038236 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-blzvl\"" Sep 17 18:29:58.044115 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.044098 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 18:29:58.098779 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.098744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-config\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:58.098954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.098796 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mx4t\" (UniqueName: \"kubernetes.io/projected/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-kube-api-access-2mx4t\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.098954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.098825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa1cff07-b71c-4c36-b79d-5376eb90acdd-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:58.098954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.098934 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgk66\" (UniqueName: \"kubernetes.io/projected/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-kube-api-access-vgk66\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:58.099062 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.098968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:58.099062 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6q29\" (UniqueName: \"kubernetes.io/projected/e1782d59-4fd9-4076-b0f4-54067c4cfd4c-kube-api-access-x6q29\") pod \"network-check-source-7cbc45fddc-6fv49\" (UID: \"e1782d59-4fd9-4076-b0f4-54067c4cfd4c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" Sep 17 18:29:58.099062 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099055 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zzz\" (UniqueName: \"kubernetes.io/projected/de0932ae-2338-4514-a5ee-9c16d66593fc-kube-api-access-q2zzz\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:58.099187 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.099260 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099234 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:58.099319 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:58.099375 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x687x\" (UniqueName: \"kubernetes.io/projected/fa1cff07-b71c-4c36-b79d-5376eb90acdd-kube-api-access-x687x\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:58.099375 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1cff07-b71c-4c36-b79d-5376eb90acdd-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:58.099504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-config\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:58.099504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099387 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-serving-cert\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:58.099504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa1cff07-b71c-4c36-b79d-5376eb90acdd-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:58.099504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099415 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-default-certificate\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.099504 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.099702 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.099528 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-stats-auth\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.101457 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.101420 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" Sep 17 18:29:58.101591 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.101573 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1cff07-b71c-4c36-b79d-5376eb90acdd-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:58.101704 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.101689 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-serving-cert\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:58.107421 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.107393 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgk66\" (UniqueName: \"kubernetes.io/projected/b08eb757-3994-4f1d-9a43-92ed2e1ca5b9-kube-api-access-vgk66\") pod \"service-ca-operator-57cdc4b768-qhbrm\" (UID: \"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:58.107641 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.107626 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x687x\" (UniqueName: \"kubernetes.io/projected/fa1cff07-b71c-4c36-b79d-5376eb90acdd-kube-api-access-x687x\") pod \"kube-storage-version-migrator-operator-55889f7b47-kt2hg\" (UID: \"fa1cff07-b71c-4c36-b79d-5376eb90acdd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:58.107696 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.107653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6q29\" (UniqueName: \"kubernetes.io/projected/e1782d59-4fd9-4076-b0f4-54067c4cfd4c-kube-api-access-x6q29\") pod \"network-check-source-7cbc45fddc-6fv49\" (UID: \"e1782d59-4fd9-4076-b0f4-54067c4cfd4c\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" Sep 17 18:29:58.113526 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.113508 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" Sep 17 18:29:58.166270 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.166239 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" Sep 17 18:29:58.189788 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.189754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" Sep 17 18:29:58.200738 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:58.200857 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200756 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:58.200857 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkv2\" (UniqueName: \"kubernetes.io/projected/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-kube-api-access-rdkv2\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.200857 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.201021 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-default-certificate\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.201021 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.201021 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f3e7043-0001-4c19-aefd-19d656d570fd-trusted-ca\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.201021 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.200977 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-stats-auth\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.201021 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7q5\" (UniqueName: \"kubernetes.io/projected/0f3e7043-0001-4c19-aefd-19d656d570fd-kube-api-access-vw7q5\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.201272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201040 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.201272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201072 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f3e7043-0001-4c19-aefd-19d656d570fd-serving-cert\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.201272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mx4t\" (UniqueName: \"kubernetes.io/projected/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-kube-api-access-2mx4t\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.201272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:58.201272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zzz\" (UniqueName: \"kubernetes.io/projected/de0932ae-2338-4514-a5ee-9c16d66593fc-kube-api-access-q2zzz\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:58.201272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201229 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.201272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201249 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f3e7043-0001-4c19-aefd-19d656d570fd-config\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.202467 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.201992 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:58.202588 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.202493 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:58.202588 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.202559 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.702540445 +0000 UTC m=+45.982191813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : secret "router-metrics-certs-default" not found Sep 17 18:29:58.202588 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.202495 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:58.202771 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.202603 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert podName:b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.702593184 +0000 UTC m=+45.982244562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hl4kd" (UID: "b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6") : secret "networking-console-plugin-cert" not found Sep 17 18:29:58.202829 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.202786 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:58.202878 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.202834 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls podName:de0932ae-2338-4514-a5ee-9c16d66593fc nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.702815302 +0000 UTC m=+45.982466668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-2ch72" (UID: "de0932ae-2338-4514-a5ee-9c16d66593fc") : secret "samples-operator-tls" not found Sep 17 18:29:58.202934 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.202884 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.702866371 +0000 UTC m=+45.982517743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:58.204737 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.204633 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-stats-auth\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.206373 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.206310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-default-certificate\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.212438 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.212406 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mx4t\" (UniqueName: \"kubernetes.io/projected/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-kube-api-access-2mx4t\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.212553 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.212539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" Sep 17 18:29:58.213653 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.213481 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zzz\" (UniqueName: \"kubernetes.io/projected/de0932ae-2338-4514-a5ee-9c16d66593fc-kube-api-access-q2zzz\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:58.302503 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302043 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.302503 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f3e7043-0001-4c19-aefd-19d656d570fd-serving-cert\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.302503 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302343 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f3e7043-0001-4c19-aefd-19d656d570fd-config\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.302503 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkv2\" (UniqueName: \"kubernetes.io/projected/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-kube-api-access-rdkv2\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.302503 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.302858 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f3e7043-0001-4c19-aefd-19d656d570fd-trusted-ca\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.302858 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7q5\" (UniqueName: \"kubernetes.io/projected/0f3e7043-0001-4c19-aefd-19d656d570fd-kube-api-access-vw7q5\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.302858 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.302696 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:58.302858 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.302767 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls podName:d4c66ecd-c5cf-4ea9-9674-edfbf632764d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.802747857 +0000 UTC m=+46.082399227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h8rn8" (UID: "d4c66ecd-c5cf-4ea9-9674-edfbf632764d") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:58.303054 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.302938 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.303147 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.303121 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f3e7043-0001-4c19-aefd-19d656d570fd-config\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.303741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.303724 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f3e7043-0001-4c19-aefd-19d656d570fd-trusted-ca\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.304605 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.304589 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f3e7043-0001-4c19-aefd-19d656d570fd-serving-cert\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.313961 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.313933 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkv2\" (UniqueName: \"kubernetes.io/projected/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-kube-api-access-rdkv2\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.314407 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.314389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7q5\" (UniqueName: \"kubernetes.io/projected/0f3e7043-0001-4c19-aefd-19d656d570fd-kube-api-access-vw7q5\") pod \"console-operator-fd4bb46c-ssmlh\" (UID: \"0f3e7043-0001-4c19-aefd-19d656d570fd\") " pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.344361 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.344321 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:29:58.407212 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.407173 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf"] Sep 17 18:29:58.410569 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:58.410542 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68273c1d_c8b7_4149_a872_0ca8a179333d.slice/crio-567e629ae4c9f01ad46606ca69108f803da52bd2b8b6f1688641550936d11c21 WatchSource:0}: Error finding container 567e629ae4c9f01ad46606ca69108f803da52bd2b8b6f1688641550936d11c21: Status 404 returned error can't find the container with id 567e629ae4c9f01ad46606ca69108f803da52bd2b8b6f1688641550936d11c21 Sep 17 18:29:58.430755 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.430723 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-x9qpw"] Sep 17 18:29:58.434831 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:58.434791 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7869cb96_52c3_46c3_b799_1412fbce679b.slice/crio-e77ad364bbb840127c99fefbf01496a4ce168b1d6e0cbe0c0dffe75a6b3ac4f0 WatchSource:0}: Error finding container e77ad364bbb840127c99fefbf01496a4ce168b1d6e0cbe0c0dffe75a6b3ac4f0: Status 404 returned error can't find the container with id e77ad364bbb840127c99fefbf01496a4ce168b1d6e0cbe0c0dffe75a6b3ac4f0 Sep 17 18:29:58.481139 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:58.481111 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1782d59_4fd9_4076_b0f4_54067c4cfd4c.slice/crio-ea979a8d18b1486770e174907bf35021868463466aaab4fda1b61b98a10ee118 WatchSource:0}: Error finding container ea979a8d18b1486770e174907bf35021868463466aaab4fda1b61b98a10ee118: Status 404 returned error can't find the container with id ea979a8d18b1486770e174907bf35021868463466aaab4fda1b61b98a10ee118 Sep 17 18:29:58.482614 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.482589 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49"] Sep 17 18:29:58.502051 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.501979 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm"] Sep 17 18:29:58.504425 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:58.504402 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08eb757_3994_4f1d_9a43_92ed2e1ca5b9.slice/crio-ae0c516473effeb48c2f3f035cfe06901bd9693cc5fc633440aefe1af107c769 WatchSource:0}: Error finding container ae0c516473effeb48c2f3f035cfe06901bd9693cc5fc633440aefe1af107c769: Status 404 returned error can't find the container with id ae0c516473effeb48c2f3f035cfe06901bd9693cc5fc633440aefe1af107c769 Sep 17 18:29:58.531260 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.531122 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg"] Sep 17 18:29:58.533607 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:58.533580 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1cff07_b71c_4c36_b79d_5376eb90acdd.slice/crio-bd013dc528a88104bb22fbd40b40d07181b6d3f22aade0cc0a3f28f4c2cfdb12 WatchSource:0}: Error finding container bd013dc528a88104bb22fbd40b40d07181b6d3f22aade0cc0a3f28f4c2cfdb12: Status 404 returned error can't find the container with id bd013dc528a88104bb22fbd40b40d07181b6d3f22aade0cc0a3f28f4c2cfdb12 Sep 17 18:29:58.645009 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.644977 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-ssmlh"] Sep 17 18:29:58.647801 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:29:58.647777 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3e7043_0001_4c19_aefd_19d656d570fd.slice/crio-6d50e333ad59b776753192ce8540e7c93f147f97945b6c8e0124c0855bb8c4ee WatchSource:0}: Error finding container 6d50e333ad59b776753192ce8540e7c93f147f97945b6c8e0124c0855bb8c4ee: Status 404 returned error can't find the container with id 6d50e333ad59b776753192ce8540e7c93f147f97945b6c8e0124c0855bb8c4ee Sep 17 18:29:58.706711 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.706673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:58.706888 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.706738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.706888 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.706799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:58.706888 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.706832 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:58.706888 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.706835 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:58.707069 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.706905 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert podName:b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.706887958 +0000 UTC m=+46.986539330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hl4kd" (UID: "b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6") : secret "networking-console-plugin-cert" not found Sep 17 18:29:58.707069 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.706920 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.706914031 +0000 UTC m=+46.986565382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:58.707069 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.706925 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:58.707069 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.706929 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:58.707069 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.706977 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.706964937 +0000 UTC m=+46.986616289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : secret "router-metrics-certs-default" not found Sep 17 18:29:58.707069 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.706990 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls podName:de0932ae-2338-4514-a5ee-9c16d66593fc nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.706984399 +0000 UTC m=+46.986635751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-2ch72" (UID: "de0932ae-2338-4514-a5ee-9c16d66593fc") : secret "samples-operator-tls" not found Sep 17 18:29:58.808395 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:58.808312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:58.808794 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.808426 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:58.808794 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:58.808500 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls podName:d4c66ecd-c5cf-4ea9-9674-edfbf632764d nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.808485622 +0000 UTC m=+47.088136987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h8rn8" (UID: "d4c66ecd-c5cf-4ea9-9674-edfbf632764d") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:59.206229 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.206176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" event={"ID":"fa1cff07-b71c-4c36-b79d-5376eb90acdd","Type":"ContainerStarted","Data":"bd013dc528a88104bb22fbd40b40d07181b6d3f22aade0cc0a3f28f4c2cfdb12"} Sep 17 18:29:59.209122 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.209041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" event={"ID":"7869cb96-52c3-46c3-b799-1412fbce679b","Type":"ContainerStarted","Data":"e77ad364bbb840127c99fefbf01496a4ce168b1d6e0cbe0c0dffe75a6b3ac4f0"} Sep 17 18:29:59.210767 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.210733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" event={"ID":"0f3e7043-0001-4c19-aefd-19d656d570fd","Type":"ContainerStarted","Data":"6d50e333ad59b776753192ce8540e7c93f147f97945b6c8e0124c0855bb8c4ee"} Sep 17 18:29:59.213113 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.213083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" event={"ID":"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9","Type":"ContainerStarted","Data":"ae0c516473effeb48c2f3f035cfe06901bd9693cc5fc633440aefe1af107c769"} Sep 17 18:29:59.215685 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.215422 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" event={"ID":"e1782d59-4fd9-4076-b0f4-54067c4cfd4c","Type":"ContainerStarted","Data":"ea979a8d18b1486770e174907bf35021868463466aaab4fda1b61b98a10ee118"} Sep 17 18:29:59.217098 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.217050 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" event={"ID":"68273c1d-c8b7-4149-a872-0ca8a179333d","Type":"ContainerStarted","Data":"567e629ae4c9f01ad46606ca69108f803da52bd2b8b6f1688641550936d11c21"} Sep 17 18:29:59.718032 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.717994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:29:59.718214 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.718070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:59.718214 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.718123 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:29:59.718214 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.718189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:29:59.718400 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.718384 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.718363579 +0000 UTC m=+48.998014945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : configmap references non-existent config key: service-ca.crt Sep 17 18:29:59.718569 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.718551 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:29:59.718624 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.718606 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.718590857 +0000 UTC m=+48.998242223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : secret "router-metrics-certs-default" not found Sep 17 18:29:59.718679 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.718665 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:29:59.718736 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.718699 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls podName:de0932ae-2338-4514-a5ee-9c16d66593fc nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.718688412 +0000 UTC m=+48.998339778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-2ch72" (UID: "de0932ae-2338-4514-a5ee-9c16d66593fc") : secret "samples-operator-tls" not found Sep 17 18:29:59.718791 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.718751 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:29:59.718791 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.718780 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert podName:b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.718771061 +0000 UTC m=+48.998422426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hl4kd" (UID: "b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6") : secret "networking-console-plugin-cert" not found Sep 17 18:29:59.820003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:29:59.819261 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:29:59.820003 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.819575 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:29:59.820003 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:29:59.819658 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls podName:d4c66ecd-c5cf-4ea9-9674-edfbf632764d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.819638234 +0000 UTC m=+49.099289605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h8rn8" (UID: "d4c66ecd-c5cf-4ea9-9674-edfbf632764d") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:30:01.739190 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:01.739151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:01.739214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:01.739278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:01.739310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.739307 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.739373 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.739393 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.739376967 +0000 UTC m=+53.019028319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : configmap references non-existent config key: service-ca.crt Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.739402 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.739413 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert podName:b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.739407104 +0000 UTC m=+53.019058456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hl4kd" (UID: "b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6") : secret "networking-console-plugin-cert" not found Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.739478 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.739465617 +0000 UTC m=+53.019116972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : secret "router-metrics-certs-default" not found Sep 17 18:30:01.739623 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.739493 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls podName:de0932ae-2338-4514-a5ee-9c16d66593fc nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.739487564 +0000 UTC m=+53.019138916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-2ch72" (UID: "de0932ae-2338-4514-a5ee-9c16d66593fc") : secret "samples-operator-tls" not found Sep 17 18:30:01.840512 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:01.840473 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:30:01.840692 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.840642 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:30:01.840752 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:01.840726 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls podName:d4c66ecd-c5cf-4ea9-9674-edfbf632764d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.840705793 +0000 UTC m=+53.120357170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h8rn8" (UID: "d4c66ecd-c5cf-4ea9-9674-edfbf632764d") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:30:04.060125 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:04.060073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:30:04.060638 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:04.060380 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:30:04.060638 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:04.060397 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68bb7dd5cf-k5s9b: secret "image-registry-tls" not found Sep 17 18:30:04.060638 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:04.060481 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls podName:d22ec93c-7ad1-4ed4-880b-e60c3faca6d3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:20.06046304 +0000 UTC m=+67.340114407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls") pod "image-registry-68bb7dd5cf-k5s9b" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3") : secret "image-registry-tls" not found Sep 17 18:30:04.160705 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:04.160666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:30:04.160906 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:04.160853 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:30:04.160971 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:04.160927 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls podName:1cc0cf8d-03e2-489e-9a01-cb24fa4ee382 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:20.160906333 +0000 UTC m=+67.440557704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls") pod "dns-default-dbkr4" (UID: "1cc0cf8d-03e2-489e-9a01-cb24fa4ee382") : secret "dns-default-metrics-tls" not found Sep 17 18:30:05.238493 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.238391 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/0.log" Sep 17 18:30:05.238493 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.238441 2568 generic.go:356] "Generic (PLEG): container finished" podID="0f3e7043-0001-4c19-aefd-19d656d570fd" containerID="c67cc5472196edcdc2dc9e237b82e82df834b6a0b69deefebef382b9661cbd05" exitCode=255 Sep 17 18:30:05.239006 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.238542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" event={"ID":"0f3e7043-0001-4c19-aefd-19d656d570fd","Type":"ContainerDied","Data":"c67cc5472196edcdc2dc9e237b82e82df834b6a0b69deefebef382b9661cbd05"} Sep 17 18:30:05.240281 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.240261 2568 scope.go:117] "RemoveContainer" containerID="c67cc5472196edcdc2dc9e237b82e82df834b6a0b69deefebef382b9661cbd05" Sep 17 18:30:05.240890 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.240264 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" event={"ID":"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9","Type":"ContainerStarted","Data":"2cfa50443c89bf3a4973b7c9861617852e46bcaebef631193e207392a8f22600"} Sep 17 18:30:05.241979 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.241956 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" event={"ID":"e1782d59-4fd9-4076-b0f4-54067c4cfd4c","Type":"ContainerStarted","Data":"7795684e83b748426c0763ff034913c30a2f8f63a8994e8b50e8ce6c614fb160"} Sep 17 18:30:05.244731 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.244705 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" event={"ID":"68273c1d-c8b7-4149-a872-0ca8a179333d","Type":"ContainerStarted","Data":"0fc916b8b49263b95728e2b95fad7a07fb94cecfad22c3b6b5da155f3dde022a"} Sep 17 18:30:05.246684 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.246665 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" event={"ID":"fa1cff07-b71c-4c36-b79d-5376eb90acdd","Type":"ContainerStarted","Data":"94e6fce55385398f377351969b91ce977b4da65d075067ced8bded6db5adb16b"} Sep 17 18:30:05.248612 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.248530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" event={"ID":"7869cb96-52c3-46c3-b799-1412fbce679b","Type":"ContainerStarted","Data":"40cb54775e6f5928f64d32e0d4674972fd922d1130059ae948ddc812b6b9b24c"} Sep 17 18:30:05.275529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.274210 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" podStartSLOduration=2.109874495 podStartE2EDuration="8.274190164s" totalImagesPullingTime="6.164315669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:05.27309559 +0000 UTC m=+52.552746965" watchObservedRunningTime="2026-09-17 18:30:05.274190164 +0000 UTC m=+52.553841542" Sep 17 18:30:05.295383 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.294587 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-b8vdf" podStartSLOduration=2.058045434 podStartE2EDuration="8.29457291s" totalImagesPullingTime="6.236527476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:05.294099273 +0000 UTC m=+52.573750650" watchObservedRunningTime="2026-09-17 18:30:05.29457291 +0000 UTC m=+52.574224287" Sep 17 18:30:05.314429 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.313773 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" podStartSLOduration=2.0794748 podStartE2EDuration="8.313750483s" totalImagesPullingTime="6.234275683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:05.313156615 +0000 UTC m=+52.592807992" watchObservedRunningTime="2026-09-17 18:30:05.313750483 +0000 UTC m=+52.593401861" Sep 17 18:30:05.334793 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.334732 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-6fv49" podStartSLOduration=1.904812854 podStartE2EDuration="8.334710083s" totalImagesPullingTime="6.429897229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:05.333498178 +0000 UTC m=+52.613149544" watchObservedRunningTime="2026-09-17 18:30:05.334710083 +0000 UTC m=+52.614361450" Sep 17 18:30:05.354152 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.353819 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" podStartSLOduration=2.213425463 podStartE2EDuration="8.353794953s" totalImagesPullingTime="6.14036949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:05.352959738 +0000 UTC m=+52.632611115" watchObservedRunningTime="2026-09-17 18:30:05.353794953 +0000 UTC m=+52.633446327" Sep 17 18:30:05.775129 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.775075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:30:05.775304 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.775172 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:05.775304 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.775246 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:30:05.775304 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.775270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:30:05.775493 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.775317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:05.775493 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.775337 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert podName:b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:13.7753159 +0000 UTC m=+61.054967258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hl4kd" (UID: "b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6") : secret "networking-console-plugin-cert" not found Sep 17 18:30:05.775493 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.775396 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:30:05.775493 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.775397 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:30:05.775493 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.775471 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:13.775429778 +0000 UTC m=+61.055081145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : secret "router-metrics-certs-default" not found Sep 17 18:30:05.775493 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.775490 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls podName:de0932ae-2338-4514-a5ee-9c16d66593fc nodeName:}" failed. No retries permitted until 2026-09-17 18:30:13.775479725 +0000 UTC m=+61.055131083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-2ch72" (UID: "de0932ae-2338-4514-a5ee-9c16d66593fc") : secret "samples-operator-tls" not found Sep 17 18:30:05.775794 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.775513 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:13.775495265 +0000 UTC m=+61.055146632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : configmap references non-existent config key: service-ca.crt Sep 17 18:30:05.876154 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:05.876108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:30:05.876354 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.876239 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:30:05.876354 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:05.876323 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls podName:d4c66ecd-c5cf-4ea9-9674-edfbf632764d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:13.876303678 +0000 UTC m=+61.155955047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h8rn8" (UID: "d4c66ecd-c5cf-4ea9-9674-edfbf632764d") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:30:06.253833 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.253804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:30:06.254280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.254263 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/0.log" Sep 17 18:30:06.254354 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.254296 2568 generic.go:356] "Generic (PLEG): container finished" podID="0f3e7043-0001-4c19-aefd-19d656d570fd" containerID="010d2d6cf2bde03d6bff76042f902aef4ff79970051a6ca4100b1b627c221731" exitCode=255 Sep 17 18:30:06.254423 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.254401 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" event={"ID":"0f3e7043-0001-4c19-aefd-19d656d570fd","Type":"ContainerDied","Data":"010d2d6cf2bde03d6bff76042f902aef4ff79970051a6ca4100b1b627c221731"} Sep 17 18:30:06.254532 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.254519 2568 scope.go:117] "RemoveContainer" containerID="c67cc5472196edcdc2dc9e237b82e82df834b6a0b69deefebef382b9661cbd05" Sep 17 18:30:06.255495 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.255473 2568 scope.go:117] "RemoveContainer" containerID="010d2d6cf2bde03d6bff76042f902aef4ff79970051a6ca4100b1b627c221731" Sep 17 18:30:06.255665 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:06.255642 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-ssmlh_openshift-console-operator(0f3e7043-0001-4c19-aefd-19d656d570fd)\"" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" podUID="0f3e7043-0001-4c19-aefd-19d656d570fd" Sep 17 18:30:06.646403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.646369 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g46wv"] Sep 17 18:30:06.676697 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.676659 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g46wv"] Sep 17 18:30:06.676891 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.676808 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.679458 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.679429 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 18:30:06.679568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.679482 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kvfvs\"" Sep 17 18:30:06.679716 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.679699 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 18:30:06.784994 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.784954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f030f12-0b17-4c45-bfb4-c428a1788ed9-data-volume\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.784994 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.785010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.785240 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.785071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.785240 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.785089 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbdg\" (UniqueName: \"kubernetes.io/projected/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-api-access-2dbdg\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.785240 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.785180 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f030f12-0b17-4c45-bfb4-c428a1788ed9-crio-socket\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.885831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.885794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f030f12-0b17-4c45-bfb4-c428a1788ed9-data-volume\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.885975 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.885838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.885975 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.885881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.885975 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.885907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbdg\" (UniqueName: \"kubernetes.io/projected/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-api-access-2dbdg\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.885975 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.885962 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f030f12-0b17-4c45-bfb4-c428a1788ed9-crio-socket\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.886185 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:06.886011 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:30:06.886185 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:06.886040 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:30:06.886185 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.886020 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f030f12-0b17-4c45-bfb4-c428a1788ed9-crio-socket\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.886185 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:06.886106 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:07.386087638 +0000 UTC m=+54.665739009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : secret "insights-runtime-extractor-tls" not found Sep 17 18:30:06.886385 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:06.886203 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:07.386189265 +0000 UTC m=+54.665840618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : configmap "kube-rbac-proxy" not found Sep 17 18:30:06.886385 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.886208 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f030f12-0b17-4c45-bfb4-c428a1788ed9-data-volume\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:06.896887 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:06.896792 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbdg\" (UniqueName: \"kubernetes.io/projected/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-api-access-2dbdg\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:07.259007 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:07.258932 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:30:07.259361 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:07.259297 2568 scope.go:117] "RemoveContainer" containerID="010d2d6cf2bde03d6bff76042f902aef4ff79970051a6ca4100b1b627c221731" Sep 17 18:30:07.259523 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:07.259501 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-ssmlh_openshift-console-operator(0f3e7043-0001-4c19-aefd-19d656d570fd)\"" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" podUID="0f3e7043-0001-4c19-aefd-19d656d570fd" Sep 17 18:30:07.391728 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:07.391693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:07.391899 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:07.391740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:07.391899 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:07.391852 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:30:07.391989 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:07.391911 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.391897669 +0000 UTC m=+55.671549034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : secret "insights-runtime-extractor-tls" not found Sep 17 18:30:07.391989 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:07.391907 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:30:07.391989 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:07.391970 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.391955229 +0000 UTC m=+55.671606581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : configmap "kube-rbac-proxy" not found Sep 17 18:30:08.186584 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:08.186556 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-75d5b_c672c626-1477-4611-a266-16e533c89d87/dns-node-resolver/0.log" Sep 17 18:30:08.344735 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:08.344694 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:30:08.345119 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:08.344790 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:30:08.345158 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:08.345128 2568 scope.go:117] "RemoveContainer" containerID="010d2d6cf2bde03d6bff76042f902aef4ff79970051a6ca4100b1b627c221731" Sep 17 18:30:08.345300 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:08.345279 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-ssmlh_openshift-console-operator(0f3e7043-0001-4c19-aefd-19d656d570fd)\"" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" podUID="0f3e7043-0001-4c19-aefd-19d656d570fd" Sep 17 18:30:08.402094 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:08.402055 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:08.402284 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:08.402115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:08.402284 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:08.402174 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:30:08.402284 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:08.402229 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.402211893 +0000 UTC m=+57.681863245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : configmap "kube-rbac-proxy" not found Sep 17 18:30:08.402284 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:08.402238 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:30:08.402508 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:08.402318 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.402300254 +0000 UTC m=+57.681951624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : secret "insights-runtime-extractor-tls" not found Sep 17 18:30:09.188886 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:09.188857 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vhft5_092bdb7d-656a-4900-8af6-f0a03eab5aa9/node-ca/0.log" Sep 17 18:30:09.988093 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:09.988054 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rnxzj_5b748973-196a-4cf2-a06f-67b2588fc4ef/serve-healthcheck-canary/0.log" Sep 17 18:30:10.420007 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:10.419962 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:10.420195 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:10.420025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:10.420195 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:10.420103 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:30:10.420195 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:10.420124 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:30:10.420195 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:10.420170 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.420150894 +0000 UTC m=+61.699802266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : configmap "kube-rbac-proxy" not found Sep 17 18:30:10.420337 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:10.420298 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.420282567 +0000 UTC m=+61.699933920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : secret "insights-runtime-extractor-tls" not found Sep 17 18:30:13.853361 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:13.853320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:13.853421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:13.853471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:13.853526 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:29.85350198 +0000 UTC m=+77.133153333 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : configmap references non-existent config key: service-ca.crt Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:13.853562 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:13.853619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:13.853677 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:13.853699 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs podName:d6f4e7bd-7144-495f-b55c-08c3a876a3ad nodeName:}" failed. No retries permitted until 2026-09-17 18:30:29.853681522 +0000 UTC m=+77.133332874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs") pod "router-default-695f977464-jzv52" (UID: "d6f4e7bd-7144-495f-b55c-08c3a876a3ad") : secret "router-metrics-certs-default" not found Sep 17 18:30:13.853834 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:13.853732 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert podName:b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:29.853719501 +0000 UTC m=+77.133370879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hl4kd" (UID: "b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6") : secret "networking-console-plugin-cert" not found Sep 17 18:30:13.855906 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:13.855888 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de0932ae-2338-4514-a5ee-9c16d66593fc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-2ch72\" (UID: \"de0932ae-2338-4514-a5ee-9c16d66593fc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:30:13.913464 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:13.913420 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" Sep 17 18:30:13.954462 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:13.954407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:30:13.954637 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:13.954574 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:30:13.954683 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:13.954665 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls podName:d4c66ecd-c5cf-4ea9-9674-edfbf632764d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:29.954643845 +0000 UTC m=+77.234295214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-h8rn8" (UID: "d4c66ecd-c5cf-4ea9-9674-edfbf632764d") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:30:14.159243 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:14.159217 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-895jp" Sep 17 18:30:14.184270 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:14.184133 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72"] Sep 17 18:30:14.282175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:14.282138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" event={"ID":"de0932ae-2338-4514-a5ee-9c16d66593fc","Type":"ContainerStarted","Data":"2cc81cf81399f1fed4a5d97cf53c1bd85bde535f40853e2423a2268e958a258b"} Sep 17 18:30:14.460018 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:14.459924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:14.460173 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:14.460052 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:30:14.460173 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:14.460126 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:22.460106266 +0000 UTC m=+69.739757637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : configmap "kube-rbac-proxy" not found Sep 17 18:30:14.460173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:14.460141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:14.460345 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:14.460234 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:30:14.460345 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:14.460268 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls podName:7f030f12-0b17-4c45-bfb4-c428a1788ed9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:22.46026009 +0000 UTC m=+69.739911443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g46wv" (UID: "7f030f12-0b17-4c45-bfb4-c428a1788ed9") : secret "insights-runtime-extractor-tls" not found Sep 17 18:30:16.291421 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:16.291381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" event={"ID":"de0932ae-2338-4514-a5ee-9c16d66593fc","Type":"ContainerStarted","Data":"6165edad97b871544433863eff858b63200a668b616d174ab41a405282250d5c"} Sep 17 18:30:16.291856 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:16.291438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" event={"ID":"de0932ae-2338-4514-a5ee-9c16d66593fc","Type":"ContainerStarted","Data":"03de40744ec164e1808ac48efe31ca12f118cf1fdfcaddbc1ab89ba850ac6c8f"} Sep 17 18:30:16.308587 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:16.308532 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-2ch72" podStartSLOduration=17.891611003 podStartE2EDuration="19.308516498s" totalImagesPullingTime="1.416905495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:16.307574379 +0000 UTC m=+63.587225754" watchObservedRunningTime="2026-09-17 18:30:16.308516498 +0000 UTC m=+63.588167871" Sep 17 18:30:20.115848 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.115806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:30:20.118226 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.118198 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"image-registry-68bb7dd5cf-k5s9b\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:30:20.217487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.217423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:30:20.220433 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.220395 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1cc0cf8d-03e2-489e-9a01-cb24fa4ee382-metrics-tls\") pod \"dns-default-dbkr4\" (UID: \"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382\") " pod="openshift-dns/dns-default-dbkr4" Sep 17 18:30:20.290553 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.290523 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fsz7t\"" Sep 17 18:30:20.298751 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.298726 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:30:20.427340 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.427309 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f4hc6\"" Sep 17 18:30:20.435835 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.435805 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbkr4" Sep 17 18:30:20.602280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.602253 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68bb7dd5cf-k5s9b"] Sep 17 18:30:20.604752 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:20.604727 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ec93c_7ad1_4ed4_880b_e60c3faca6d3.slice/crio-527041be7a3f5907aad893a771ad08bea7e53591a55766e8e55a6a0b63660fa2 WatchSource:0}: Error finding container 527041be7a3f5907aad893a771ad08bea7e53591a55766e8e55a6a0b63660fa2: Status 404 returned error can't find the container with id 527041be7a3f5907aad893a771ad08bea7e53591a55766e8e55a6a0b63660fa2 Sep 17 18:30:20.630408 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.630318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:30:20.632967 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.632946 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 18:30:20.643418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.643389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8747e254-9790-4236-87a0-9955cb416dde-original-pull-secret\") pod \"global-pull-secret-syncer-6hvl9\" (UID: \"8747e254-9790-4236-87a0-9955cb416dde\") " pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:30:20.731844 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.731801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:30:20.732026 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.731859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:30:20.734611 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.734586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb4hp\" (UniqueName: \"kubernetes.io/projected/00162636-5bc2-435a-8fbe-39f28f63641d-kube-api-access-mb4hp\") pod \"network-check-target-wh8zt\" (UID: \"00162636-5bc2-435a-8fbe-39f28f63641d\") " pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:30:20.734708 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.734631 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 18:30:20.742725 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.742700 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbkr4"] Sep 17 18:30:20.744384 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.744360 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/370e5c7b-6a22-41a2-bbe8-67e7f11bc373-metrics-certs\") pod \"network-metrics-daemon-zq6bz\" (UID: \"370e5c7b-6a22-41a2-bbe8-67e7f11bc373\") " pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:30:20.745877 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:20.745855 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc0cf8d_03e2_489e_9a01_cb24fa4ee382.slice/crio-659fa6fe6f88968e98b52ce0578a600556ffcfd7fe54f027f403f4a368e1f6da WatchSource:0}: Error finding container 659fa6fe6f88968e98b52ce0578a600556ffcfd7fe54f027f403f4a368e1f6da: Status 404 returned error can't find the container with id 659fa6fe6f88968e98b52ce0578a600556ffcfd7fe54f027f403f4a368e1f6da Sep 17 18:30:20.791387 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.791358 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhngp\"" Sep 17 18:30:20.799417 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.799386 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:30:20.892228 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.892151 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmb9h\"" Sep 17 18:30:20.894194 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.894176 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hvl9" Sep 17 18:30:20.900306 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:20.900285 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zq6bz" Sep 17 18:30:21.082511 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.082478 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wh8zt"] Sep 17 18:30:21.085292 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:21.085262 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00162636_5bc2_435a_8fbe_39f28f63641d.slice/crio-ebec4eb3ec14b04fd1c646608e6322ee3bb07855a2625c55818105c6d782e4d9 WatchSource:0}: Error finding container ebec4eb3ec14b04fd1c646608e6322ee3bb07855a2625c55818105c6d782e4d9: Status 404 returned error can't find the container with id ebec4eb3ec14b04fd1c646608e6322ee3bb07855a2625c55818105c6d782e4d9 Sep 17 18:30:21.181754 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.181719 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hvl9"] Sep 17 18:30:21.183998 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.183973 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zq6bz"] Sep 17 18:30:21.185285 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:21.185250 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8747e254_9790_4236_87a0_9955cb416dde.slice/crio-7728d47a72d8bc938c0df7342c6156cf5db147985003d973a0a39298c5ca8d41 WatchSource:0}: Error finding container 7728d47a72d8bc938c0df7342c6156cf5db147985003d973a0a39298c5ca8d41: Status 404 returned error can't find the container with id 7728d47a72d8bc938c0df7342c6156cf5db147985003d973a0a39298c5ca8d41 Sep 17 18:30:21.187589 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:21.187561 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370e5c7b_6a22_41a2_bbe8_67e7f11bc373.slice/crio-9e734adab5df1b4537b2d661192c2b03d29aea59a23ddbb9613d7807fdcade06 WatchSource:0}: Error finding container 9e734adab5df1b4537b2d661192c2b03d29aea59a23ddbb9613d7807fdcade06: Status 404 returned error can't find the container with id 9e734adab5df1b4537b2d661192c2b03d29aea59a23ddbb9613d7807fdcade06 Sep 17 18:30:21.308816 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.308773 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" event={"ID":"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3","Type":"ContainerStarted","Data":"e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8"} Sep 17 18:30:21.309020 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.308835 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" event={"ID":"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3","Type":"ContainerStarted","Data":"527041be7a3f5907aad893a771ad08bea7e53591a55766e8e55a6a0b63660fa2"} Sep 17 18:30:21.309020 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.308871 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:30:21.310743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.310632 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wh8zt" event={"ID":"00162636-5bc2-435a-8fbe-39f28f63641d","Type":"ContainerStarted","Data":"5f357c0191fe2373da925bda064777e50a7217a2a1319a4e37402d3c87440331"} Sep 17 18:30:21.310743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.310690 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wh8zt" event={"ID":"00162636-5bc2-435a-8fbe-39f28f63641d","Type":"ContainerStarted","Data":"ebec4eb3ec14b04fd1c646608e6322ee3bb07855a2625c55818105c6d782e4d9"} Sep 17 18:30:21.310743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.310717 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:30:21.312230 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.312193 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq6bz" event={"ID":"370e5c7b-6a22-41a2-bbe8-67e7f11bc373","Type":"ContainerStarted","Data":"9e734adab5df1b4537b2d661192c2b03d29aea59a23ddbb9613d7807fdcade06"} Sep 17 18:30:21.313694 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.313674 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbkr4" event={"ID":"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382","Type":"ContainerStarted","Data":"659fa6fe6f88968e98b52ce0578a600556ffcfd7fe54f027f403f4a368e1f6da"} Sep 17 18:30:21.314970 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.314946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hvl9" event={"ID":"8747e254-9790-4236-87a0-9955cb416dde","Type":"ContainerStarted","Data":"7728d47a72d8bc938c0df7342c6156cf5db147985003d973a0a39298c5ca8d41"} Sep 17 18:30:21.332955 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.332904 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" podStartSLOduration=66.332889196 podStartE2EDuration="1m6.332889196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:21.332063754 +0000 UTC m=+68.611715131" watchObservedRunningTime="2026-09-17 18:30:21.332889196 +0000 UTC m=+68.612540568" Sep 17 18:30:21.358376 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.358323 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wh8zt" podStartSLOduration=66.358305533 podStartE2EDuration="1m6.358305533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:21.357721842 +0000 UTC m=+68.637373220" watchObservedRunningTime="2026-09-17 18:30:21.358305533 +0000 UTC m=+68.637956907" Sep 17 18:30:21.979746 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:21.979504 2568 scope.go:117] "RemoveContainer" containerID="010d2d6cf2bde03d6bff76042f902aef4ff79970051a6ca4100b1b627c221731" Sep 17 18:30:22.550565 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:22.550530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:22.550952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:22.550584 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:22.551129 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:22.551109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f030f12-0b17-4c45-bfb4-c428a1788ed9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:22.553322 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:22.553280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f030f12-0b17-4c45-bfb4-c428a1788ed9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g46wv\" (UID: \"7f030f12-0b17-4c45-bfb4-c428a1788ed9\") " pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:22.588880 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:22.588844 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kvfvs\"" Sep 17 18:30:22.596411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:22.596378 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g46wv" Sep 17 18:30:23.328124 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.327995 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:30:23.328124 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.328107 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" event={"ID":"0f3e7043-0001-4c19-aefd-19d656d570fd","Type":"ContainerStarted","Data":"f4cbd017d5f194e41d06b095b660d26f5d69fff10c0866c6b427401219378801"} Sep 17 18:30:23.329112 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.329063 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:30:23.331993 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.331925 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq6bz" event={"ID":"370e5c7b-6a22-41a2-bbe8-67e7f11bc373","Type":"ContainerStarted","Data":"8e305645a9d351b8f371fe171d66850177038c8b1a837d2fe1b4c1da5338c4b0"} Sep 17 18:30:23.334356 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.334332 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbkr4" event={"ID":"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382","Type":"ContainerStarted","Data":"2b20a81ba54f138f93287b7855f3881099e266d030cdfc150abdc15e0a7c33dc"} Sep 17 18:30:23.349784 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.349738 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" podStartSLOduration=20.328709937 podStartE2EDuration="26.349717558s" totalImagesPullingTime="6.021007621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:23.348252189 +0000 UTC m=+70.627903564" watchObservedRunningTime="2026-09-17 18:30:23.349717558 +0000 UTC m=+70.629368934" Sep 17 18:30:23.388948 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.388828 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g46wv"] Sep 17 18:30:23.392954 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:23.392926 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f030f12_0b17_4c45_bfb4_c428a1788ed9.slice/crio-42c3c6ee0b85a858715ec9c23b971b816d1d18dcf5989b37156a74b3b2a022e2 WatchSource:0}: Error finding container 42c3c6ee0b85a858715ec9c23b971b816d1d18dcf5989b37156a74b3b2a022e2: Status 404 returned error can't find the container with id 42c3c6ee0b85a858715ec9c23b971b816d1d18dcf5989b37156a74b3b2a022e2 Sep 17 18:30:23.461560 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:23.461537 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-ssmlh" Sep 17 18:30:24.342428 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:24.342394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g46wv" event={"ID":"7f030f12-0b17-4c45-bfb4-c428a1788ed9","Type":"ContainerStarted","Data":"66215e2663a11f395638f5ed5fdcfe3a0f68ac9f6b41206cefd67d8068e3368f"} Sep 17 18:30:24.342909 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:24.342478 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g46wv" event={"ID":"7f030f12-0b17-4c45-bfb4-c428a1788ed9","Type":"ContainerStarted","Data":"42c3c6ee0b85a858715ec9c23b971b816d1d18dcf5989b37156a74b3b2a022e2"} Sep 17 18:30:24.344500 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:24.344464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zq6bz" event={"ID":"370e5c7b-6a22-41a2-bbe8-67e7f11bc373","Type":"ContainerStarted","Data":"28dffd9eb6229e9972004258908be5dc147e8ae9b8f1c75f38fa7be77281ada3"} Sep 17 18:30:24.346501 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:24.346476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbkr4" event={"ID":"1cc0cf8d-03e2-489e-9a01-cb24fa4ee382","Type":"ContainerStarted","Data":"cf391e0bb09dee9305ebea9779a770e546f6fa576a5e8f9cd85fa23eba644e08"} Sep 17 18:30:24.346619 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:24.346598 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbkr4" Sep 17 18:30:24.369664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:24.369624 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zq6bz" podStartSLOduration=67.457732139 podStartE2EDuration="1m9.369606349s" totalImagesPullingTime="1.91187421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:24.367071624 +0000 UTC m=+71.646723000" watchObservedRunningTime="2026-09-17 18:30:24.369606349 +0000 UTC m=+71.649257723" Sep 17 18:30:24.389962 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:24.389911 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dbkr4" podStartSLOduration=34.035977327 podStartE2EDuration="36.389891985s" totalImagesPullingTime="2.353914658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:24.387259454 +0000 UTC m=+71.666910829" watchObservedRunningTime="2026-09-17 18:30:24.389891985 +0000 UTC m=+71.669543358" Sep 17 18:30:25.352242 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:25.352213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hvl9" event={"ID":"8747e254-9790-4236-87a0-9955cb416dde","Type":"ContainerStarted","Data":"590c3532b45fff49657a96b401c2084bd937edda56b0092f802808111724348c"} Sep 17 18:30:25.374122 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:25.374027 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6hvl9" podStartSLOduration=66.449458114 podStartE2EDuration="1m10.374004199s" totalImagesPullingTime="3.924546085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:25.372164439 +0000 UTC m=+72.651815827" watchObservedRunningTime="2026-09-17 18:30:25.374004199 +0000 UTC m=+72.653655574" Sep 17 18:30:26.358787 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:26.358744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g46wv" event={"ID":"7f030f12-0b17-4c45-bfb4-c428a1788ed9","Type":"ContainerStarted","Data":"7cd92ec987653564ae2d3b167c3bbf5357a293f97aa4995c69fb45f7df8e3c08"} Sep 17 18:30:27.363932 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:27.363895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g46wv" event={"ID":"7f030f12-0b17-4c45-bfb4-c428a1788ed9","Type":"ContainerStarted","Data":"49e6e7d8334ecee369252b124a4c79639640bd6b1cda5ec0c384102ed525780c"} Sep 17 18:30:27.385008 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:27.384954 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g46wv" podStartSLOduration=18.353241051 podStartE2EDuration="21.384937643s" totalImagesPullingTime="3.031696592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:27.3838609 +0000 UTC m=+74.663512275" watchObservedRunningTime="2026-09-17 18:30:27.384937643 +0000 UTC m=+74.664589017" Sep 17 18:30:29.920199 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:29.920163 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:29.920609 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:29.920251 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:29.920609 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:29.920280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:30:29.920884 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:29.920848 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-service-ca-bundle\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:29.922751 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:29.922722 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6f4e7bd-7144-495f-b55c-08c3a876a3ad-metrics-certs\") pod \"router-default-695f977464-jzv52\" (UID: \"d6f4e7bd-7144-495f-b55c-08c3a876a3ad\") " pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:29.922857 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:29.922753 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hl4kd\" (UID: \"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:30:30.020824 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.020787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:30:30.022325 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.022309 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tb84v\"" Sep 17 18:30:30.023162 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.023141 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4c66ecd-c5cf-4ea9-9674-edfbf632764d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-h8rn8\" (UID: \"d4c66ecd-c5cf-4ea9-9674-edfbf632764d\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:30:30.029669 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.029652 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:30.088995 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.088964 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-79kvt\"" Sep 17 18:30:30.095847 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.095775 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" Sep 17 18:30:30.127115 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.127087 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nj488\"" Sep 17 18:30:30.134500 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.134472 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" Sep 17 18:30:30.315192 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.315159 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-695f977464-jzv52"] Sep 17 18:30:30.318090 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:30.318062 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f4e7bd_7144_495f_b55c_08c3a876a3ad.slice/crio-f4f3b38fc09090984843133b4eb95dd778bba620e6dcb47f9957e29a34f9c3ef WatchSource:0}: Error finding container f4f3b38fc09090984843133b4eb95dd778bba620e6dcb47f9957e29a34f9c3ef: Status 404 returned error can't find the container with id f4f3b38fc09090984843133b4eb95dd778bba620e6dcb47f9957e29a34f9c3ef Sep 17 18:30:30.371283 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.371253 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd"] Sep 17 18:30:30.374552 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:30.374517 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f7c3c5_0275_4c94_a7d7_0c7bc64806b6.slice/crio-97e301d7e6cf43accbf51bbeac54db501b3f250d36b3568e86d05147566cba57 WatchSource:0}: Error finding container 97e301d7e6cf43accbf51bbeac54db501b3f250d36b3568e86d05147566cba57: Status 404 returned error can't find the container with id 97e301d7e6cf43accbf51bbeac54db501b3f250d36b3568e86d05147566cba57 Sep 17 18:30:30.375806 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.375778 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-695f977464-jzv52" event={"ID":"d6f4e7bd-7144-495f-b55c-08c3a876a3ad","Type":"ContainerStarted","Data":"f4f3b38fc09090984843133b4eb95dd778bba620e6dcb47f9957e29a34f9c3ef"} Sep 17 18:30:30.420027 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:30.419997 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8"] Sep 17 18:30:30.422951 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:30.422927 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c66ecd_c5cf_4ea9_9674_edfbf632764d.slice/crio-5862e25d8b8b45e5941e34af0a7ff74a406468dfd91262641289188dc79eea74 WatchSource:0}: Error finding container 5862e25d8b8b45e5941e34af0a7ff74a406468dfd91262641289188dc79eea74: Status 404 returned error can't find the container with id 5862e25d8b8b45e5941e34af0a7ff74a406468dfd91262641289188dc79eea74 Sep 17 18:30:31.381263 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:31.381220 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" event={"ID":"d4c66ecd-c5cf-4ea9-9674-edfbf632764d","Type":"ContainerStarted","Data":"5862e25d8b8b45e5941e34af0a7ff74a406468dfd91262641289188dc79eea74"} Sep 17 18:30:31.382793 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:31.382758 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" event={"ID":"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6","Type":"ContainerStarted","Data":"97e301d7e6cf43accbf51bbeac54db501b3f250d36b3568e86d05147566cba57"} Sep 17 18:30:31.384566 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:31.384538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-695f977464-jzv52" event={"ID":"d6f4e7bd-7144-495f-b55c-08c3a876a3ad","Type":"ContainerStarted","Data":"f488263b8b3c366c23f91c8601adf78be858e092304bc6855332a97ac1d359c3"} Sep 17 18:30:31.429038 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:31.428981 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-695f977464-jzv52" podStartSLOduration=34.428964817 podStartE2EDuration="34.428964817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:31.427905275 +0000 UTC m=+78.707556662" watchObservedRunningTime="2026-09-17 18:30:31.428964817 +0000 UTC m=+78.708616212" Sep 17 18:30:32.030529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:32.030491 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:32.033355 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:32.033327 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:32.390516 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:32.390478 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" event={"ID":"b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6","Type":"ContainerStarted","Data":"af7738a9453a849554b9a24f86f7a94611b3619001ea163a1e230220c3b2b5db"} Sep 17 18:30:32.390978 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:32.390780 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:32.392175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:32.392155 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-695f977464-jzv52" Sep 17 18:30:32.408554 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:32.408499 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hl4kd" podStartSLOduration=34.322933095 podStartE2EDuration="35.408476746s" totalImagesPullingTime="1.085543651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:32.40725082 +0000 UTC m=+79.686902198" watchObservedRunningTime="2026-09-17 18:30:32.408476746 +0000 UTC m=+79.688128123" Sep 17 18:30:33.395911 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:33.395871 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" event={"ID":"d4c66ecd-c5cf-4ea9-9674-edfbf632764d","Type":"ContainerStarted","Data":"163a27e315b9812235a160509a9d478453111512a82f8c0d73bcdf0679226876"} Sep 17 18:30:33.414179 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:33.414116 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-h8rn8" podStartSLOduration=34.267276734 podStartE2EDuration="36.414092513s" totalImagesPullingTime="2.146815779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:33.413009023 +0000 UTC m=+80.692660398" watchObservedRunningTime="2026-09-17 18:30:33.414092513 +0000 UTC m=+80.693743890" Sep 17 18:30:34.355568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:34.355535 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbkr4" Sep 17 18:30:36.896559 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.896523 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f684c9d7f-grb7v"] Sep 17 18:30:36.926655 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.926614 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f684c9d7f-grb7v"] Sep 17 18:30:36.926840 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.926714 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:36.929254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.929215 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 18:30:36.929254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.929230 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 18:30:36.929254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.929250 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 18:30:36.929629 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.929609 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hdv42\"" Sep 17 18:30:36.929976 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.929954 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 18:30:36.930080 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.930037 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 18:30:36.930080 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.930052 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 18:30:36.930186 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.930175 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 18:30:36.977323 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.977283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-config\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:36.977502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.977332 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-oauth-serving-cert\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:36.977502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.977365 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbws7\" (UniqueName: \"kubernetes.io/projected/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-kube-api-access-fbws7\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:36.977502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.977406 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-serving-cert\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:36.977502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.977484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-oauth-config\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:36.977638 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:36.977504 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-service-ca\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.078668 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.078630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-config\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.078850 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.078681 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-oauth-serving-cert\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.078850 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.078702 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbws7\" (UniqueName: \"kubernetes.io/projected/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-kube-api-access-fbws7\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.078850 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.078743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-serving-cert\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.078850 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.078767 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-oauth-config\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.078850 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.078781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-service-ca\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.079550 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.079522 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-oauth-serving-cert\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.079550 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.079547 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-config\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.079733 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.079528 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-service-ca\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.081143 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.081119 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-oauth-config\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.081240 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.081184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-serving-cert\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.090073 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.090046 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbws7\" (UniqueName: \"kubernetes.io/projected/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-kube-api-access-fbws7\") pod \"console-7f684c9d7f-grb7v\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.235696 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.235606 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:37.533875 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:37.533803 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f684c9d7f-grb7v"] Sep 17 18:30:37.537001 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:37.536976 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1c13ce_b6e2_453d_a8a0_7d06b43b9d80.slice/crio-ea44c4f1260ac7eb0bda9b494f1eb319b023813e8fc54912cf5e24ef8d3c427c WatchSource:0}: Error finding container ea44c4f1260ac7eb0bda9b494f1eb319b023813e8fc54912cf5e24ef8d3c427c: Status 404 returned error can't find the container with id ea44c4f1260ac7eb0bda9b494f1eb319b023813e8fc54912cf5e24ef8d3c427c Sep 17 18:30:38.418888 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:38.418784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f684c9d7f-grb7v" event={"ID":"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80","Type":"ContainerStarted","Data":"ea44c4f1260ac7eb0bda9b494f1eb319b023813e8fc54912cf5e24ef8d3c427c"} Sep 17 18:30:42.325848 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:42.325823 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:30:42.441544 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:42.441474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f684c9d7f-grb7v" event={"ID":"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80","Type":"ContainerStarted","Data":"db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01"} Sep 17 18:30:42.468315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:42.468268 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f684c9d7f-grb7v" podStartSLOduration=2.59098972 podStartE2EDuration="6.468250913s" totalImagesPullingTime="3.877261193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:42.467346931 +0000 UTC m=+89.746998305" watchObservedRunningTime="2026-09-17 18:30:42.468250913 +0000 UTC m=+89.747902283" Sep 17 18:30:43.843737 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.843702 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-t2lzq"] Sep 17 18:30:43.846071 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.846051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:43.857281 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.857257 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 17 18:30:43.857487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.857470 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 17 18:30:43.858523 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.858504 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 18:30:43.858616 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.858583 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 17 18:30:43.858747 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.858725 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9npdf\"" Sep 17 18:30:43.891010 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.890983 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-t2lzq"] Sep 17 18:30:43.892392 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.892363 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vqjr6"] Sep 17 18:30:43.894665 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.894647 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.897205 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.897186 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 18:30:43.897303 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.897288 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-csfkx\"" Sep 17 18:30:43.897831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.897818 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 18:30:43.898285 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.898264 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 18:30:43.945389 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:43.945517 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945400 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-wtmp\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945517 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945424 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:43.945517 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945456 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-sys\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945517 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-textfile\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945517 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9d9b\" (UniqueName: \"kubernetes.io/projected/fe52b640-325a-4086-93ad-8d0829be056a-kube-api-access-n9d9b\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945678 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbcc\" (UniqueName: \"kubernetes.io/projected/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-api-access-hpbcc\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:43.945678 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-tls\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945678 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945678 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-root\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945678 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:43.945820 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945688 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe52b640-325a-4086-93ad-8d0829be056a-metrics-client-ca\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:43.945820 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:43.945820 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945776 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:43.945820 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:43.945798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.046633 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-root\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.046766 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046637 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.046766 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046658 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe52b640-325a-4086-93ad-8d0829be056a-metrics-client-ca\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.046766 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046675 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.046766 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.046766 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.046766 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046730 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-root\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.047079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046894 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-wtmp\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046934 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.047079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-sys\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.046992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-textfile\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9d9b\" (UniqueName: \"kubernetes.io/projected/fe52b640-325a-4086-93ad-8d0829be056a-kube-api-access-n9d9b\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbcc\" (UniqueName: \"kubernetes.io/projected/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-api-access-hpbcc\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.047411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-tls\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047142 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.047411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe52b640-325a-4086-93ad-8d0829be056a-metrics-client-ca\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-wtmp\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047441 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.047651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047596 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.047651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe52b640-325a-4086-93ad-8d0829be056a-sys\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047847 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:44.047685 2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 17 18:30:44.047847 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047708 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047847 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.047714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-textfile\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.047847 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:30:44.047753 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-tls podName:3f93f2df-5aba-4ee6-9125-bb7370fdc5c4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:44.547735838 +0000 UTC m=+91.827387204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-t2lzq" (UID: "3f93f2df-5aba-4ee6-9125-bb7370fdc5c4") : secret "kube-state-metrics-tls" not found Sep 17 18:30:44.049701 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.049673 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-tls\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.049826 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.049809 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe52b640-325a-4086-93ad-8d0829be056a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.050003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.049983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.056325 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.056304 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbcc\" (UniqueName: \"kubernetes.io/projected/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-api-access-hpbcc\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.058031 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.058012 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9d9b\" (UniqueName: \"kubernetes.io/projected/fe52b640-325a-4086-93ad-8d0829be056a-kube-api-access-n9d9b\") pod \"node-exporter-vqjr6\" (UID: \"fe52b640-325a-4086-93ad-8d0829be056a\") " pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.203639 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.203603 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vqjr6" Sep 17 18:30:44.213394 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:44.213361 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe52b640_325a_4086_93ad_8d0829be056a.slice/crio-7a0a03b3a1b7138d4e2b858db7f2306cf2f9838ff85e77875f00ad473f660582 WatchSource:0}: Error finding container 7a0a03b3a1b7138d4e2b858db7f2306cf2f9838ff85e77875f00ad473f660582: Status 404 returned error can't find the container with id 7a0a03b3a1b7138d4e2b858db7f2306cf2f9838ff85e77875f00ad473f660582 Sep 17 18:30:44.449942 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.449904 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqjr6" event={"ID":"fe52b640-325a-4086-93ad-8d0829be056a","Type":"ContainerStarted","Data":"7a0a03b3a1b7138d4e2b858db7f2306cf2f9838ff85e77875f00ad473f660582"} Sep 17 18:30:44.551923 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.551831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.554126 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.554099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f93f2df-5aba-4ee6-9125-bb7370fdc5c4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-t2lzq\" (UID: \"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:44.754597 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:44.754554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" Sep 17 18:30:45.040600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:45.040570 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-t2lzq"] Sep 17 18:30:45.042504 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:45.042480 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f93f2df_5aba_4ee6_9125_bb7370fdc5c4.slice/crio-2337b99a31567271be32ce554727900df170e84a441e99522f9649d971007402 WatchSource:0}: Error finding container 2337b99a31567271be32ce554727900df170e84a441e99522f9649d971007402: Status 404 returned error can't find the container with id 2337b99a31567271be32ce554727900df170e84a441e99522f9649d971007402 Sep 17 18:30:45.455789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:45.455748 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" event={"ID":"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4","Type":"ContainerStarted","Data":"2337b99a31567271be32ce554727900df170e84a441e99522f9649d971007402"} Sep 17 18:30:45.457461 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:45.457421 2568 generic.go:356] "Generic (PLEG): container finished" podID="fe52b640-325a-4086-93ad-8d0829be056a" containerID="3c81f7a37164b5ed4f251ae6b23b94ec97d213324b10039edfa8fffdf209bc71" exitCode=0 Sep 17 18:30:45.457583 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:45.457503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqjr6" event={"ID":"fe52b640-325a-4086-93ad-8d0829be056a","Type":"ContainerDied","Data":"3c81f7a37164b5ed4f251ae6b23b94ec97d213324b10039edfa8fffdf209bc71"} Sep 17 18:30:46.467362 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.467244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" event={"ID":"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4","Type":"ContainerStarted","Data":"6a95f86d49b11f8c9eeb702b12858f42572bcb5f5f84f0d72c1ce92c610d3f3d"} Sep 17 18:30:46.467362 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.467323 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" event={"ID":"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4","Type":"ContainerStarted","Data":"e5a05d53db6fe2887eefa54f5d6e61fd4542849cf74a3b4689b15ac182084b3e"} Sep 17 18:30:46.467362 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.467348 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" event={"ID":"3f93f2df-5aba-4ee6-9125-bb7370fdc5c4","Type":"ContainerStarted","Data":"de859c71c978790cb210fb54142ffe67002a3f42fbf6aa769750746228825463"} Sep 17 18:30:46.469693 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.469666 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqjr6" event={"ID":"fe52b640-325a-4086-93ad-8d0829be056a","Type":"ContainerStarted","Data":"38c8e1846df6727484bfb3322d51234d4b8e4927ef4233bd7aab62deab9cd789"} Sep 17 18:30:46.469796 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.469711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vqjr6" event={"ID":"fe52b640-325a-4086-93ad-8d0829be056a","Type":"ContainerStarted","Data":"44d42f7b0feb7cd5434659212df10c6b8881a4a38f8abca8cf9c0541be441d5c"} Sep 17 18:30:46.508204 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.508142 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vqjr6" podStartSLOduration=2.617455074 podStartE2EDuration="3.508124108s" totalImagesPullingTime="820.954705ms" totalInitContainerRuntime="69.714329ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:46.507483208 +0000 UTC m=+93.787134585" watchObservedRunningTime="2026-09-17 18:30:46.508124108 +0000 UTC m=+93.787775483" Sep 17 18:30:46.509007 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.508973 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-t2lzq" podStartSLOduration=2.407911031 podStartE2EDuration="3.508960054s" totalImagesPullingTime="1.101049023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:46.487752161 +0000 UTC m=+93.767403534" watchObservedRunningTime="2026-09-17 18:30:46.508960054 +0000 UTC m=+93.788611428" Sep 17 18:30:46.655010 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.654976 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68c6f75df8-fjmnm"] Sep 17 18:30:46.657710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.657691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.660770 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.660335 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 17 18:30:46.660770 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.660599 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 17 18:30:46.660770 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.660685 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 17 18:30:46.661100 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.661083 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 17 18:30:46.661272 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.661240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 17 18:30:46.661362 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.661320 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8drsd11q4c5mr\"" Sep 17 18:30:46.661488 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.661439 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9hvt4\"" Sep 17 18:30:46.676266 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.676234 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68c6f75df8-fjmnm"] Sep 17 18:30:46.773041 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.772964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.773041 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.773017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-tls\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.773220 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.773068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-metrics-client-ca\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.773220 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.773115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbh2\" (UniqueName: \"kubernetes.io/projected/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-kube-api-access-6cbh2\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.773220 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.773153 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.773220 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.773176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.773220 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.773209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-grpc-tls\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.773367 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.773226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874597 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-tls\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-metrics-client-ca\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbh2\" (UniqueName: \"kubernetes.io/projected/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-kube-api-access-6cbh2\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-grpc-tls\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.874805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.874745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.875691 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.875643 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-metrics-client-ca\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.877417 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.877384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.877547 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.877519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.877765 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.877748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.877883 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.877862 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.877978 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.877958 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-grpc-tls\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.878017 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.877986 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-secret-thanos-querier-tls\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.883976 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.883953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbh2\" (UniqueName: \"kubernetes.io/projected/8683d8bc-3ce6-4dfc-be72-d86c2890ba21-kube-api-access-6cbh2\") pod \"thanos-querier-68c6f75df8-fjmnm\" (UID: \"8683d8bc-3ce6-4dfc-be72-d86c2890ba21\") " pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:46.969023 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:46.968983 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:47.235990 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:47.235947 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:47.236159 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:47.236014 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:47.240989 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:47.240964 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:47.253792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:47.253764 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68c6f75df8-fjmnm"] Sep 17 18:30:47.257908 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:47.257884 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8683d8bc_3ce6_4dfc_be72_d86c2890ba21.slice/crio-6d9fdf12b4feb9b874917cab0906cb21774c846b89b523006f288f47300ed4c5 WatchSource:0}: Error finding container 6d9fdf12b4feb9b874917cab0906cb21774c846b89b523006f288f47300ed4c5: Status 404 returned error can't find the container with id 6d9fdf12b4feb9b874917cab0906cb21774c846b89b523006f288f47300ed4c5 Sep 17 18:30:47.474431 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:47.474390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" event={"ID":"8683d8bc-3ce6-4dfc-be72-d86c2890ba21","Type":"ContainerStarted","Data":"6d9fdf12b4feb9b874917cab0906cb21774c846b89b523006f288f47300ed4c5"} Sep 17 18:30:47.478391 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:47.478366 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:30:48.306991 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.306954 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd"] Sep 17 18:30:48.309559 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.309523 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.312363 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.312336 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 17 18:30:48.312526 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.312337 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-54fnf\"" Sep 17 18:30:48.324949 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.324920 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd"] Sep 17 18:30:48.390195 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.390159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a84c96cf-3f47-49d7-bd1e-0651f3169664-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-p4tpd\" (UID: \"a84c96cf-3f47-49d7-bd1e-0651f3169664\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.390386 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.390286 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlsj\" (UniqueName: \"kubernetes.io/projected/a84c96cf-3f47-49d7-bd1e-0651f3169664-kube-api-access-4wlsj\") pod \"monitoring-plugin-5984459ff8-p4tpd\" (UID: \"a84c96cf-3f47-49d7-bd1e-0651f3169664\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.491698 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.491662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a84c96cf-3f47-49d7-bd1e-0651f3169664-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-p4tpd\" (UID: \"a84c96cf-3f47-49d7-bd1e-0651f3169664\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.492173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.491882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlsj\" (UniqueName: \"kubernetes.io/projected/a84c96cf-3f47-49d7-bd1e-0651f3169664-kube-api-access-4wlsj\") pod \"monitoring-plugin-5984459ff8-p4tpd\" (UID: \"a84c96cf-3f47-49d7-bd1e-0651f3169664\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.494701 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.494669 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a84c96cf-3f47-49d7-bd1e-0651f3169664-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-p4tpd\" (UID: \"a84c96cf-3f47-49d7-bd1e-0651f3169664\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.503745 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.503718 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlsj\" (UniqueName: \"kubernetes.io/projected/a84c96cf-3f47-49d7-bd1e-0651f3169664-kube-api-access-4wlsj\") pod \"monitoring-plugin-5984459ff8-p4tpd\" (UID: \"a84c96cf-3f47-49d7-bd1e-0651f3169664\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.622756 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.622724 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:48.911639 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:48.911604 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd"] Sep 17 18:30:49.257963 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:49.257863 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84c96cf_3f47_49d7_bd1e_0651f3169664.slice/crio-de0a796d3729949e60d47fceb48b07cd83c579be186faf25dd9717a71c38b744 WatchSource:0}: Error finding container de0a796d3729949e60d47fceb48b07cd83c579be186faf25dd9717a71c38b744: Status 404 returned error can't find the container with id de0a796d3729949e60d47fceb48b07cd83c579be186faf25dd9717a71c38b744 Sep 17 18:30:49.485148 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:49.485116 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" event={"ID":"8683d8bc-3ce6-4dfc-be72-d86c2890ba21","Type":"ContainerStarted","Data":"0c1d82bdcb24141c558fb26b2040bb54e45820f1c9abd0a0ca04c381d3a8b84b"} Sep 17 18:30:49.486306 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:49.486285 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" event={"ID":"a84c96cf-3f47-49d7-bd1e-0651f3169664","Type":"ContainerStarted","Data":"de0a796d3729949e60d47fceb48b07cd83c579be186faf25dd9717a71c38b744"} Sep 17 18:30:50.004821 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.004784 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:30:50.009026 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.009001 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.015875 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.015847 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 18:30:50.016214 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.016194 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ll2j8mrptpea\"" Sep 17 18:30:50.018432 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.018386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 18:30:50.018699 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.018675 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 18:30:50.018881 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.018854 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 18:30:50.018986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.018891 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 18:30:50.018986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.018944 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 18:30:50.018986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.018968 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 18:30:50.018986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.018895 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 18:30:50.019192 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.019171 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nlfrz\"" Sep 17 18:30:50.019274 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.019250 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 18:30:50.019414 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.019396 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 18:30:50.019602 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.019581 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 18:30:50.026491 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.026440 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 18:30:50.029912 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.029849 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 18:30:50.030099 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.030078 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 18:30:50.031325 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.031293 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 18:30:50.055834 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.055730 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:30:50.108816 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.108780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-config-out\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.108816 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.108819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.108849 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.108966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-web-config\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109145 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109145 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109225 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109225 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109305 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109255 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109353 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109413 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109413 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109393 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-config\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109558 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109471 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109558 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109558 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhx78\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-kube-api-access-hhx78\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109688 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109559 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109688 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109688 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.109814 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.109711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.169605 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.169564 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c4c575c9-nwzxq"] Sep 17 18:30:50.172969 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.172938 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.207769 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.207736 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 18:30:50.210822 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.210792 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-config\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.210980 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.210846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-oauth-config\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.210980 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.210885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.210980 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.210913 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.210980 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.210936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhx78\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-kube-api-access-hhx78\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211277 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211248 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6pl\" (UniqueName: \"kubernetes.io/projected/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-kube-api-access-4n6pl\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.211374 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211306 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211374 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211332 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-trusted-ca-bundle\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.211374 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211357 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211554 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211383 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-service-ca\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.211554 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211554 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211554 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-config\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.211554 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-config-out\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211557 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-serving-cert\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211664 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-web-config\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211721 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211747 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-oauth-serving-cert\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211776 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.211807 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211832 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211911 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.211916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.212004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.212095 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212547 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.212388 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.212547 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.212404 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.213094 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.213065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.214054 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.214028 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-config\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.214054 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.214359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.214593 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.214541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.216184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.214755 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.216184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.215469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.216184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.215821 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.216184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.215863 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.216184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.216142 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.216692 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.216647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-config-out\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.217076 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.217056 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.217271 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.217250 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-web-config\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.218137 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.218114 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.218630 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.218603 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.219253 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.219228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.243619 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.243580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhx78\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-kube-api-access-hhx78\") pod \"prometheus-k8s-0\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.280096 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.280006 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c4c575c9-nwzxq"] Sep 17 18:30:50.313355 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.313309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6pl\" (UniqueName: \"kubernetes.io/projected/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-kube-api-access-4n6pl\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.313607 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.313373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-trusted-ca-bundle\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.313607 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.313402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-service-ca\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.313607 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.313439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-config\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.313607 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.313486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-serving-cert\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.313607 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.313537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-oauth-serving-cert\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.313881 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.313623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-oauth-config\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.314501 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.314424 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-config\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.314501 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.314491 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-trusted-ca-bundle\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.314780 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.314755 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-oauth-serving-cert\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.315172 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.315145 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-service-ca\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.316720 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.316701 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-serving-cert\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.316803 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.316727 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-oauth-config\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.322224 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.322195 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:30:50.323951 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.323929 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6pl\" (UniqueName: \"kubernetes.io/projected/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-kube-api-access-4n6pl\") pod \"console-78c4c575c9-nwzxq\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.483654 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.483616 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:30:50.493576 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.493534 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" event={"ID":"8683d8bc-3ce6-4dfc-be72-d86c2890ba21","Type":"ContainerStarted","Data":"02aacafd7c1235ec8d05f7e6d93cbd38dfade270e8d7a926fb98127c78e89337"} Sep 17 18:30:50.493779 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.493623 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" event={"ID":"8683d8bc-3ce6-4dfc-be72-d86c2890ba21","Type":"ContainerStarted","Data":"dc29ad0f0d406e681df0ebba36a27ca585d4d0fd0a1afa382984d8b2cf463bc4"} Sep 17 18:30:50.923384 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.923307 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c4c575c9-nwzxq"] Sep 17 18:30:50.926711 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:50.926679 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715c5d7b_d406_4def_bbe4_7e6ea1b95afd.slice/crio-fa06f393b2892a640ff1c680c2cb3deab2ea39990bed774795cf19c006b61613 WatchSource:0}: Error finding container fa06f393b2892a640ff1c680c2cb3deab2ea39990bed774795cf19c006b61613: Status 404 returned error can't find the container with id fa06f393b2892a640ff1c680c2cb3deab2ea39990bed774795cf19c006b61613 Sep 17 18:30:50.942686 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:50.942662 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:30:50.945330 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:30:50.945301 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70848e8_df6f_4708_afc5_143655dcc679.slice/crio-f87a299701814219a69490c9794d88a9c2f0d1f444a6978a1747d9b7cab56bfa WatchSource:0}: Error finding container f87a299701814219a69490c9794d88a9c2f0d1f444a6978a1747d9b7cab56bfa: Status 404 returned error can't find the container with id f87a299701814219a69490c9794d88a9c2f0d1f444a6978a1747d9b7cab56bfa Sep 17 18:30:51.501653 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.501563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" event={"ID":"a84c96cf-3f47-49d7-bd1e-0651f3169664","Type":"ContainerStarted","Data":"65c2f6c8956ab41ce442943cb2a5b3fb75b754a9adee21531559dffa1fcf350f"} Sep 17 18:30:51.502087 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.501895 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:51.503900 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.503868 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c4c575c9-nwzxq" event={"ID":"715c5d7b-d406-4def-bbe4-7e6ea1b95afd","Type":"ContainerStarted","Data":"e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3"} Sep 17 18:30:51.504050 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.503918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c4c575c9-nwzxq" event={"ID":"715c5d7b-d406-4def-bbe4-7e6ea1b95afd","Type":"ContainerStarted","Data":"fa06f393b2892a640ff1c680c2cb3deab2ea39990bed774795cf19c006b61613"} Sep 17 18:30:51.507912 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.507889 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" Sep 17 18:30:51.508805 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.508784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" event={"ID":"8683d8bc-3ce6-4dfc-be72-d86c2890ba21","Type":"ContainerStarted","Data":"0e996fb8667a905728b8e33e447f2be0e65738a3e679a03609acadb9ec2cbeeb"} Sep 17 18:30:51.508897 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.508849 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" event={"ID":"8683d8bc-3ce6-4dfc-be72-d86c2890ba21","Type":"ContainerStarted","Data":"3d749c7a088ef23c455555ae85d9dd094610d5749560dfcbe07a9b11b84f0f94"} Sep 17 18:30:51.508897 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.508893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" event={"ID":"8683d8bc-3ce6-4dfc-be72-d86c2890ba21","Type":"ContainerStarted","Data":"2f8340fa0305006c9491e287850a92d32fbdfdbe5a109654aa8be9b8b0c1e39e"} Sep 17 18:30:51.509000 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.508940 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:30:51.510210 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.510185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerStarted","Data":"f87a299701814219a69490c9794d88a9c2f0d1f444a6978a1747d9b7cab56bfa"} Sep 17 18:30:51.527636 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.527595 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-p4tpd" podStartSLOduration=2.144661458 podStartE2EDuration="3.527579744s" totalImagesPullingTime="1.382918286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:51.527509697 +0000 UTC m=+98.807161075" watchObservedRunningTime="2026-09-17 18:30:51.527579744 +0000 UTC m=+98.807231119" Sep 17 18:30:51.564651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.563326 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" podStartSLOduration=2.397811943 podStartE2EDuration="5.563305328s" totalImagesPullingTime="3.165493385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:51.560182521 +0000 UTC m=+98.839833896" watchObservedRunningTime="2026-09-17 18:30:51.563305328 +0000 UTC m=+98.842956706" Sep 17 18:30:51.620045 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:51.619956 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78c4c575c9-nwzxq" podStartSLOduration=1.619934213 podStartE2EDuration="1.619934213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:51.619376077 +0000 UTC m=+98.899027451" watchObservedRunningTime="2026-09-17 18:30:51.619934213 +0000 UTC m=+98.899585589" Sep 17 18:30:52.324229 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:52.324200 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wh8zt" Sep 17 18:30:52.516386 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:52.516289 2568 generic.go:356] "Generic (PLEG): container finished" podID="f70848e8-df6f-4708-afc5-143655dcc679" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" exitCode=0 Sep 17 18:30:52.516878 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:52.516379 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f"} Sep 17 18:30:53.621856 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:53.621820 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68bb7dd5cf-k5s9b"] Sep 17 18:30:56.538352 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:56.538319 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerStarted","Data":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} Sep 17 18:30:56.538741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:56.538385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerStarted","Data":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} Sep 17 18:30:56.538741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:56.538412 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerStarted","Data":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} Sep 17 18:30:56.538741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:56.538441 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerStarted","Data":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} Sep 17 18:30:56.538741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:56.538483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerStarted","Data":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} Sep 17 18:30:56.538741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:56.538507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerStarted","Data":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} Sep 17 18:30:56.582626 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:56.582558 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.677768878 podStartE2EDuration="7.582535645s" totalImagesPullingTime="3.886002253s" totalInitContainerRuntime="18.764514ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:56.579909935 +0000 UTC m=+103.859561310" watchObservedRunningTime="2026-09-17 18:30:56.582535645 +0000 UTC m=+103.862187019" Sep 17 18:30:57.522785 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:30:57.522754 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68c6f75df8-fjmnm" Sep 17 18:31:00.323369 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:00.323321 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:00.484549 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:00.484510 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:31:00.484745 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:00.484577 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:31:00.489580 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:00.489554 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:31:00.561201 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:00.561170 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:31:00.621151 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:00.621120 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f684c9d7f-grb7v"] Sep 17 18:31:11.617783 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:11.617746 2568 generic.go:356] "Generic (PLEG): container finished" podID="fa1cff07-b71c-4c36-b79d-5376eb90acdd" containerID="94e6fce55385398f377351969b91ce977b4da65d075067ced8bded6db5adb16b" exitCode=0 Sep 17 18:31:11.618187 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:11.617824 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" event={"ID":"fa1cff07-b71c-4c36-b79d-5376eb90acdd","Type":"ContainerDied","Data":"94e6fce55385398f377351969b91ce977b4da65d075067ced8bded6db5adb16b"} Sep 17 18:31:11.618187 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:11.618138 2568 scope.go:117] "RemoveContainer" containerID="94e6fce55385398f377351969b91ce977b4da65d075067ced8bded6db5adb16b" Sep 17 18:31:12.624725 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:12.624685 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kt2hg" event={"ID":"fa1cff07-b71c-4c36-b79d-5376eb90acdd","Type":"ContainerStarted","Data":"d0afb3380266a7a30cd86186d72df6726e53a3b89579cfeb394664f81802248f"} Sep 17 18:31:16.643118 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:16.643086 2568 generic.go:356] "Generic (PLEG): container finished" podID="7869cb96-52c3-46c3-b799-1412fbce679b" containerID="40cb54775e6f5928f64d32e0d4674972fd922d1130059ae948ddc812b6b9b24c" exitCode=0 Sep 17 18:31:16.643495 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:16.643146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" event={"ID":"7869cb96-52c3-46c3-b799-1412fbce679b","Type":"ContainerDied","Data":"40cb54775e6f5928f64d32e0d4674972fd922d1130059ae948ddc812b6b9b24c"} Sep 17 18:31:16.643566 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:16.643544 2568 scope.go:117] "RemoveContainer" containerID="40cb54775e6f5928f64d32e0d4674972fd922d1130059ae948ddc812b6b9b24c" Sep 17 18:31:17.285269 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:17.285233 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-695f977464-jzv52_d6f4e7bd-7144-495f-b55c-08c3a876a3ad/router/0.log" Sep 17 18:31:17.308779 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:17.308736 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rnxzj_5b748973-196a-4cf2-a06f-67b2588fc4ef/serve-healthcheck-canary/0.log" Sep 17 18:31:17.650266 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:17.650230 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-x9qpw" event={"ID":"7869cb96-52c3-46c3-b799-1412fbce679b","Type":"ContainerStarted","Data":"73f1118fbc7011b7330b1cab5317d61db1cd49a40ecbf0adb90939ed0c967e83"} Sep 17 18:31:18.649773 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:18.649721 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" podUID="d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" containerName="registry" containerID="cri-o://e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8" gracePeriod=30 Sep 17 18:31:18.900889 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:18.900828 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:31:19.085157 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085116 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-trusted-ca\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085157 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085163 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-certificates\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085405 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085181 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggnpb\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-kube-api-access-ggnpb\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085405 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085210 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-bound-sa-token\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085405 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085326 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-installation-pull-secrets\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085405 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085392 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085427 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-image-registry-private-configuration\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085500 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-ca-trust-extracted\") pod \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\" (UID: \"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3\") " Sep 17 18:31:19.085651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085633 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:19.085810 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085715 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:19.085876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085820 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-trusted-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.085876 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.085838 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-certificates\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.087922 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.087885 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:19.088139 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.088103 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:19.088254 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.088147 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:19.088310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.088250 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-kube-api-access-ggnpb" (OuterVolumeSpecName: "kube-api-access-ggnpb") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "kube-api-access-ggnpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:19.088310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.088273 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:19.093575 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.093551 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" (UID: "d22ec93c-7ad1-4ed4-880b-e60c3faca6d3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:19.186586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.186505 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-bound-sa-token\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.186586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.186535 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-installation-pull-secrets\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.186586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.186546 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-registry-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.186586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.186556 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-image-registry-private-configuration\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.186586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.186565 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-ca-trust-extracted\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.186586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.186574 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggnpb\" (UniqueName: \"kubernetes.io/projected/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3-kube-api-access-ggnpb\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:19.660049 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.660015 2568 generic.go:356] "Generic (PLEG): container finished" podID="d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" containerID="e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8" exitCode=0 Sep 17 18:31:19.660300 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.660064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" event={"ID":"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3","Type":"ContainerDied","Data":"e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8"} Sep 17 18:31:19.660300 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.660074 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" Sep 17 18:31:19.660300 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.660106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68bb7dd5cf-k5s9b" event={"ID":"d22ec93c-7ad1-4ed4-880b-e60c3faca6d3","Type":"ContainerDied","Data":"527041be7a3f5907aad893a771ad08bea7e53591a55766e8e55a6a0b63660fa2"} Sep 17 18:31:19.660300 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.660126 2568 scope.go:117] "RemoveContainer" containerID="e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8" Sep 17 18:31:19.668770 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.668751 2568 scope.go:117] "RemoveContainer" containerID="e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8" Sep 17 18:31:19.669057 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:31:19.669036 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8\": container with ID starting with e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8 not found: ID does not exist" containerID="e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8" Sep 17 18:31:19.669103 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.669065 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8"} err="failed to get container status \"e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8\": rpc error: code = NotFound desc = could not find container \"e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8\": container with ID starting with e0da40cfa4f77ca3b8576f8dcab2e3b65bea2751aeb12834a9e841c8b93d26e8 not found: ID does not exist" Sep 17 18:31:19.681269 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.681243 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68bb7dd5cf-k5s9b"] Sep 17 18:31:19.685214 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:19.685192 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68bb7dd5cf-k5s9b"] Sep 17 18:31:20.984083 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:20.984047 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" path="/var/lib/kubelet/pods/d22ec93c-7ad1-4ed4-880b-e60c3faca6d3/volumes" Sep 17 18:31:25.642238 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.642197 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f684c9d7f-grb7v" podUID="9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" containerName="console" containerID="cri-o://db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01" gracePeriod=15 Sep 17 18:31:25.688219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.688190 2568 generic.go:356] "Generic (PLEG): container finished" podID="b08eb757-3994-4f1d-9a43-92ed2e1ca5b9" containerID="2cfa50443c89bf3a4973b7c9861617852e46bcaebef631193e207392a8f22600" exitCode=0 Sep 17 18:31:25.688356 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.688266 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" event={"ID":"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9","Type":"ContainerDied","Data":"2cfa50443c89bf3a4973b7c9861617852e46bcaebef631193e207392a8f22600"} Sep 17 18:31:25.688681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.688663 2568 scope.go:117] "RemoveContainer" containerID="2cfa50443c89bf3a4973b7c9861617852e46bcaebef631193e207392a8f22600" Sep 17 18:31:25.895979 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.895916 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:31:25.943474 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.943412 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-config\") pod \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " Sep 17 18:31:25.943667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.943470 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-serving-cert\") pod \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " Sep 17 18:31:25.943667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.943507 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-service-ca\") pod \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " Sep 17 18:31:25.943667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.943559 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-oauth-config\") pod \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " Sep 17 18:31:25.943667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.943594 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-oauth-serving-cert\") pod \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " Sep 17 18:31:25.943667 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.943670 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbws7\" (UniqueName: \"kubernetes.io/projected/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-kube-api-access-fbws7\") pod \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\" (UID: \"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80\") " Sep 17 18:31:25.944182 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.943966 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-config" (OuterVolumeSpecName: "console-config") pod "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" (UID: "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:25.944182 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.944055 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" (UID: "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:25.944338 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.944205 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" (UID: "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:31:25.946396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.946373 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" (UID: "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:25.946500 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.946388 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" (UID: "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:25.946567 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:25.946551 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-kube-api-access-fbws7" (OuterVolumeSpecName: "kube-api-access-fbws7") pod "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" (UID: "9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80"). InnerVolumeSpecName "kube-api-access-fbws7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:26.045033 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.044999 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-service-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:26.045033 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.045029 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-oauth-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:26.045033 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.045038 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-oauth-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:26.045242 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.045047 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbws7\" (UniqueName: \"kubernetes.io/projected/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-kube-api-access-fbws7\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:26.045242 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.045056 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:26.045242 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.045064 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80-console-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:31:26.694473 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.694423 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-qhbrm" event={"ID":"b08eb757-3994-4f1d-9a43-92ed2e1ca5b9","Type":"ContainerStarted","Data":"4dd4e13ea0c41f44eb530f3948108eaf3bbb84ab8cf9f504d465333a1d5c038d"} Sep 17 18:31:26.695764 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.695741 2568 generic.go:356] "Generic (PLEG): container finished" podID="9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" containerID="db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01" exitCode=0 Sep 17 18:31:26.695887 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.695798 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f684c9d7f-grb7v" Sep 17 18:31:26.695887 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.695822 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f684c9d7f-grb7v" event={"ID":"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80","Type":"ContainerDied","Data":"db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01"} Sep 17 18:31:26.695887 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.695873 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f684c9d7f-grb7v" event={"ID":"9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80","Type":"ContainerDied","Data":"ea44c4f1260ac7eb0bda9b494f1eb319b023813e8fc54912cf5e24ef8d3c427c"} Sep 17 18:31:26.696034 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.695897 2568 scope.go:117] "RemoveContainer" containerID="db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01" Sep 17 18:31:26.709605 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.709582 2568 scope.go:117] "RemoveContainer" containerID="db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01" Sep 17 18:31:26.709938 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:31:26.709919 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01\": container with ID starting with db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01 not found: ID does not exist" containerID="db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01" Sep 17 18:31:26.709987 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.709948 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01"} err="failed to get container status \"db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01\": rpc error: code = NotFound desc = could not find container \"db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01\": container with ID starting with db8610d34333d2e053d87f8c5b057a59433323128f5aeb37b203a7d38bee5a01 not found: ID does not exist" Sep 17 18:31:26.734267 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.734229 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f684c9d7f-grb7v"] Sep 17 18:31:26.737883 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.737852 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f684c9d7f-grb7v"] Sep 17 18:31:26.984406 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:26.984323 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" path="/var/lib/kubelet/pods/9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80/volumes" Sep 17 18:31:50.322881 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:50.322814 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:50.338935 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:50.338910 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:31:50.834138 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:31:50.834110 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:08.390359 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.390271 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:32:08.390789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.390594 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="prometheus" containerID="cri-o://5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" gracePeriod=600 Sep 17 18:32:08.390789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.390603 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy" containerID="cri-o://902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" gracePeriod=600 Sep 17 18:32:08.390789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.390631 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-web" containerID="cri-o://4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" gracePeriod=600 Sep 17 18:32:08.390789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.390670 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" gracePeriod=600 Sep 17 18:32:08.390789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.390682 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="config-reloader" containerID="cri-o://0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" gracePeriod=600 Sep 17 18:32:08.391024 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.390626 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="thanos-sidecar" containerID="cri-o://916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" gracePeriod=600 Sep 17 18:32:08.636342 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.636315 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:08.826776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826674 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-kubelet-serving-ca-bundle\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.826776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826721 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-tls-assets\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.826776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826748 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-grpc-tls\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.826776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826775 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826802 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-config-out\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826833 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-serving-certs-ca-bundle\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826876 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-config\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826915 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-1\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.826970 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhx78\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-kube-api-access-hhx78\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827001 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-metrics-client-ca\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827029 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-thanos-prometheus-http-client-file\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827056 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-trusted-ca-bundle\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827105 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827094 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-db\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827128 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-tls\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827149 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827160 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827223 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-2\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827237 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827254 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-kube-rbac-proxy\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827288 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-0\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827322 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-web-config\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.827600 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827352 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-metrics-client-certs\") pod \"f70848e8-df6f-4708-afc5-143655dcc679\" (UID: \"f70848e8-df6f-4708-afc5-143655dcc679\") " Sep 17 18:32:08.828073 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827683 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.828073 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.827705 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.829502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829474 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.829614 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829531 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:32:08.829614 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829529 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.829856 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829836 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:08.829946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829849 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-config" (OuterVolumeSpecName: "config") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.829946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829851 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-kube-api-access-hhx78" (OuterVolumeSpecName: "kube-api-access-hhx78") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "kube-api-access-hhx78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:32:08.829946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829882 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.829946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.829916 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:08.830491 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.830199 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:08.830491 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.830321 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:08.830787 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.830725 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.830787 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.830731 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:32:08.831009 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.830981 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-config-out" (OuterVolumeSpecName: "config-out") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:32:08.831401 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.831380 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:08.831939 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.831915 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.832159 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.832140 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.832223 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.832207 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.842931 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.842910 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-web-config" (OuterVolumeSpecName: "web-config") pod "f70848e8-df6f-4708-afc5-143655dcc679" (UID: "f70848e8-df6f-4708-afc5-143655dcc679"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:08.900462 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900420 2568 generic.go:356] "Generic (PLEG): container finished" podID="f70848e8-df6f-4708-afc5-143655dcc679" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" exitCode=0 Sep 17 18:32:08.900462 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900459 2568 generic.go:356] "Generic (PLEG): container finished" podID="f70848e8-df6f-4708-afc5-143655dcc679" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" exitCode=0 Sep 17 18:32:08.900462 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900468 2568 generic.go:356] "Generic (PLEG): container finished" podID="f70848e8-df6f-4708-afc5-143655dcc679" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" exitCode=0 Sep 17 18:32:08.900664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900475 2568 generic.go:356] "Generic (PLEG): container finished" podID="f70848e8-df6f-4708-afc5-143655dcc679" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" exitCode=0 Sep 17 18:32:08.900664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900480 2568 generic.go:356] "Generic (PLEG): container finished" podID="f70848e8-df6f-4708-afc5-143655dcc679" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" exitCode=0 Sep 17 18:32:08.900664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900485 2568 generic.go:356] "Generic (PLEG): container finished" podID="f70848e8-df6f-4708-afc5-143655dcc679" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" exitCode=0 Sep 17 18:32:08.900664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900511 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} Sep 17 18:32:08.900664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900530 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:08.900664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900615 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} Sep 17 18:32:08.900664 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} Sep 17 18:32:08.900870 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900669 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} Sep 17 18:32:08.900870 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900680 2568 scope.go:117] "RemoveContainer" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.900870 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900697 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} Sep 17 18:32:08.900870 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} Sep 17 18:32:08.900870 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.900751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f70848e8-df6f-4708-afc5-143655dcc679","Type":"ContainerDied","Data":"f87a299701814219a69490c9794d88a9c2f0d1f444a6978a1747d9b7cab56bfa"} Sep 17 18:32:08.909708 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.909691 2568 scope.go:117] "RemoveContainer" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.917638 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.917622 2568 scope.go:117] "RemoveContainer" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.924375 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.924353 2568 scope.go:117] "RemoveContainer" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.928587 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928565 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhx78\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-kube-api-access-hhx78\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928591 2568 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-configmap-metrics-client-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928604 2568 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928613 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928627 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-db\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928639 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928648 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928657 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928666 2568 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-kube-rbac-proxy\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928674 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.928681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928684 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-web-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.929175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928692 2568 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-metrics-client-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.929175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928705 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f70848e8-df6f-4708-afc5-143655dcc679-tls-assets\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.929175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928715 2568 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-grpc-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.929175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928725 2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.929175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928737 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f70848e8-df6f-4708-afc5-143655dcc679-config-out\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.929175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928750 2568 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f70848e8-df6f-4708-afc5-143655dcc679-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.929175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.928761 2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f70848e8-df6f-4708-afc5-143655dcc679-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:08.931558 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.931532 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:32:08.932591 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.932534 2568 scope.go:117] "RemoveContainer" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.933355 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.933336 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:32:08.939632 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.939616 2568 scope.go:117] "RemoveContainer" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.947284 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.947267 2568 scope.go:117] "RemoveContainer" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.954743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.954727 2568 scope.go:117] "RemoveContainer" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.955013 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:08.954998 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": container with ID starting with c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111 not found: ID does not exist" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.955066 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955022 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} err="failed to get container status \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": rpc error: code = NotFound desc = could not find container \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": container with ID starting with c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111 not found: ID does not exist" Sep 17 18:32:08.955066 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955040 2568 scope.go:117] "RemoveContainer" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.955273 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:08.955257 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": container with ID starting with 902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589 not found: ID does not exist" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.955310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955276 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} err="failed to get container status \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": rpc error: code = NotFound desc = could not find container \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": container with ID starting with 902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589 not found: ID does not exist" Sep 17 18:32:08.955310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955289 2568 scope.go:117] "RemoveContainer" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.955555 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:08.955539 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": container with ID starting with 4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5 not found: ID does not exist" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.955616 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955559 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} err="failed to get container status \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": rpc error: code = NotFound desc = could not find container \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": container with ID starting with 4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5 not found: ID does not exist" Sep 17 18:32:08.955616 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955572 2568 scope.go:117] "RemoveContainer" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.955820 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:08.955803 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": container with ID starting with 916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63 not found: ID does not exist" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.955861 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955826 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} err="failed to get container status \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": rpc error: code = NotFound desc = could not find container \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": container with ID starting with 916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63 not found: ID does not exist" Sep 17 18:32:08.955861 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.955843 2568 scope.go:117] "RemoveContainer" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.956073 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:08.956054 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": container with ID starting with 0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00 not found: ID does not exist" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.956163 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956073 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} err="failed to get container status \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": rpc error: code = NotFound desc = could not find container \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": container with ID starting with 0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00 not found: ID does not exist" Sep 17 18:32:08.956163 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956085 2568 scope.go:117] "RemoveContainer" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.956291 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:08.956277 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": container with ID starting with 5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e not found: ID does not exist" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.956337 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956293 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} err="failed to get container status \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": rpc error: code = NotFound desc = could not find container \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": container with ID starting with 5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e not found: ID does not exist" Sep 17 18:32:08.956337 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956304 2568 scope.go:117] "RemoveContainer" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.956534 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:08.956515 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": container with ID starting with 2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f not found: ID does not exist" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.956617 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956536 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f"} err="failed to get container status \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": rpc error: code = NotFound desc = could not find container \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": container with ID starting with 2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f not found: ID does not exist" Sep 17 18:32:08.956617 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956549 2568 scope.go:117] "RemoveContainer" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.956789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956771 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} err="failed to get container status \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": rpc error: code = NotFound desc = could not find container \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": container with ID starting with c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111 not found: ID does not exist" Sep 17 18:32:08.956829 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.956790 2568 scope.go:117] "RemoveContainer" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.957047 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957028 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} err="failed to get container status \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": rpc error: code = NotFound desc = could not find container \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": container with ID starting with 902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589 not found: ID does not exist" Sep 17 18:32:08.957088 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957055 2568 scope.go:117] "RemoveContainer" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.957284 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957268 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} err="failed to get container status \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": rpc error: code = NotFound desc = could not find container \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": container with ID starting with 4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5 not found: ID does not exist" Sep 17 18:32:08.957284 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957282 2568 scope.go:117] "RemoveContainer" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.957526 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957508 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} err="failed to get container status \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": rpc error: code = NotFound desc = could not find container \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": container with ID starting with 916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63 not found: ID does not exist" Sep 17 18:32:08.957568 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957526 2568 scope.go:117] "RemoveContainer" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.957752 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957734 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} err="failed to get container status \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": rpc error: code = NotFound desc = could not find container \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": container with ID starting with 0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00 not found: ID does not exist" Sep 17 18:32:08.957804 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957754 2568 scope.go:117] "RemoveContainer" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.957979 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957963 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} err="failed to get container status \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": rpc error: code = NotFound desc = could not find container \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": container with ID starting with 5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e not found: ID does not exist" Sep 17 18:32:08.958020 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.957979 2568 scope.go:117] "RemoveContainer" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.958183 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958168 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f"} err="failed to get container status \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": rpc error: code = NotFound desc = could not find container \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": container with ID starting with 2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f not found: ID does not exist" Sep 17 18:32:08.958234 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958182 2568 scope.go:117] "RemoveContainer" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.958407 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958389 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} err="failed to get container status \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": rpc error: code = NotFound desc = could not find container \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": container with ID starting with c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111 not found: ID does not exist" Sep 17 18:32:08.958472 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958408 2568 scope.go:117] "RemoveContainer" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.958677 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958661 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} err="failed to get container status \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": rpc error: code = NotFound desc = could not find container \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": container with ID starting with 902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589 not found: ID does not exist" Sep 17 18:32:08.958726 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958678 2568 scope.go:117] "RemoveContainer" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.958916 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958897 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} err="failed to get container status \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": rpc error: code = NotFound desc = could not find container \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": container with ID starting with 4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5 not found: ID does not exist" Sep 17 18:32:08.958965 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.958916 2568 scope.go:117] "RemoveContainer" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.959140 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.959120 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} err="failed to get container status \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": rpc error: code = NotFound desc = could not find container \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": container with ID starting with 916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63 not found: ID does not exist" Sep 17 18:32:08.959140 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.959137 2568 scope.go:117] "RemoveContainer" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.959363 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.959346 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} err="failed to get container status \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": rpc error: code = NotFound desc = could not find container \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": container with ID starting with 0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00 not found: ID does not exist" Sep 17 18:32:08.959363 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.959362 2568 scope.go:117] "RemoveContainer" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.963157 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.963121 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} err="failed to get container status \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": rpc error: code = NotFound desc = could not find container \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": container with ID starting with 5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e not found: ID does not exist" Sep 17 18:32:08.963157 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.963147 2568 scope.go:117] "RemoveContainer" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.963786 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.963751 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f"} err="failed to get container status \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": rpc error: code = NotFound desc = could not find container \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": container with ID starting with 2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f not found: ID does not exist" Sep 17 18:32:08.963786 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.963785 2568 scope.go:117] "RemoveContainer" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.964059 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964039 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} err="failed to get container status \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": rpc error: code = NotFound desc = could not find container \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": container with ID starting with c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111 not found: ID does not exist" Sep 17 18:32:08.964132 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964059 2568 scope.go:117] "RemoveContainer" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.964320 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964301 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} err="failed to get container status \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": rpc error: code = NotFound desc = could not find container \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": container with ID starting with 902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589 not found: ID does not exist" Sep 17 18:32:08.964413 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964348 2568 scope.go:117] "RemoveContainer" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.964622 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964593 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} err="failed to get container status \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": rpc error: code = NotFound desc = could not find container \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": container with ID starting with 4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5 not found: ID does not exist" Sep 17 18:32:08.964694 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964622 2568 scope.go:117] "RemoveContainer" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.964993 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964974 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} err="failed to get container status \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": rpc error: code = NotFound desc = could not find container \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": container with ID starting with 916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63 not found: ID does not exist" Sep 17 18:32:08.965099 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.964996 2568 scope.go:117] "RemoveContainer" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.965304 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965173 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:32:08.965304 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965213 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} err="failed to get container status \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": rpc error: code = NotFound desc = could not find container \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": container with ID starting with 0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00 not found: ID does not exist" Sep 17 18:32:08.965304 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965230 2568 scope.go:117] "RemoveContainer" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.965480 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965412 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} err="failed to get container status \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": rpc error: code = NotFound desc = could not find container \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": container with ID starting with 5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e not found: ID does not exist" Sep 17 18:32:08.965480 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965426 2568 scope.go:117] "RemoveContainer" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.965684 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965667 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" containerName="registry" Sep 17 18:32:08.965741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965687 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" containerName="registry" Sep 17 18:32:08.965741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965710 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="prometheus" Sep 17 18:32:08.965741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965719 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="prometheus" Sep 17 18:32:08.965741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965731 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" containerName="console" Sep 17 18:32:08.965741 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965740 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" containerName="console" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965751 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-web" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965760 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-web" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965780 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965684 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f"} err="failed to get container status \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": rpc error: code = NotFound desc = could not find container \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": container with ID starting with 2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f not found: ID does not exist" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965799 2568 scope.go:117] "RemoveContainer" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965789 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965838 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-thanos" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965846 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-thanos" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965861 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="config-reloader" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965867 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="config-reloader" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965873 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="thanos-sidecar" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965879 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="thanos-sidecar" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965890 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="init-config-reloader" Sep 17 18:32:08.965946 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965897 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="init-config-reloader" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965974 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} err="failed to get container status \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": rpc error: code = NotFound desc = could not find container \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": container with ID starting with c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111 not found: ID does not exist" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965999 2568 scope.go:117] "RemoveContainer" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.965985 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b1c13ce-b6e2-453d-a8a0-7d06b43b9d80" containerName="console" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966039 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-web" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966049 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="prometheus" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966056 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966065 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="kube-rbac-proxy-thanos" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966074 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22ec93c-7ad1-4ed4-880b-e60c3faca6d3" containerName="registry" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966082 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="thanos-sidecar" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966087 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70848e8-df6f-4708-afc5-143655dcc679" containerName="config-reloader" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966191 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} err="failed to get container status \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": rpc error: code = NotFound desc = could not find container \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": container with ID starting with 902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589 not found: ID does not exist" Sep 17 18:32:08.966403 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966204 2568 scope.go:117] "RemoveContainer" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.966952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966437 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} err="failed to get container status \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": rpc error: code = NotFound desc = could not find container \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": container with ID starting with 4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5 not found: ID does not exist" Sep 17 18:32:08.966952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966470 2568 scope.go:117] "RemoveContainer" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.966952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966698 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} err="failed to get container status \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": rpc error: code = NotFound desc = could not find container \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": container with ID starting with 916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63 not found: ID does not exist" Sep 17 18:32:08.966952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966720 2568 scope.go:117] "RemoveContainer" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.967106 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966968 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} err="failed to get container status \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": rpc error: code = NotFound desc = could not find container \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": container with ID starting with 0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00 not found: ID does not exist" Sep 17 18:32:08.967106 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.966981 2568 scope.go:117] "RemoveContainer" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.967224 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967209 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} err="failed to get container status \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": rpc error: code = NotFound desc = could not find container \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": container with ID starting with 5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e not found: ID does not exist" Sep 17 18:32:08.967262 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967224 2568 scope.go:117] "RemoveContainer" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.967430 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967413 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f"} err="failed to get container status \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": rpc error: code = NotFound desc = could not find container \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": container with ID starting with 2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f not found: ID does not exist" Sep 17 18:32:08.967502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967429 2568 scope.go:117] "RemoveContainer" containerID="c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111" Sep 17 18:32:08.967683 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967666 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111"} err="failed to get container status \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": rpc error: code = NotFound desc = could not find container \"c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111\": container with ID starting with c5ac000498193d1499ed883b453c690826c3a787f11ca9700f60d72b89f5e111 not found: ID does not exist" Sep 17 18:32:08.967726 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967683 2568 scope.go:117] "RemoveContainer" containerID="902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589" Sep 17 18:32:08.967863 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967847 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589"} err="failed to get container status \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": rpc error: code = NotFound desc = could not find container \"902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589\": container with ID starting with 902e4a4172434b83a18cbd6febc7536d153233ff0f19b170d7701077a359a589 not found: ID does not exist" Sep 17 18:32:08.967918 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.967865 2568 scope.go:117] "RemoveContainer" containerID="4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5" Sep 17 18:32:08.968034 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968018 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5"} err="failed to get container status \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": rpc error: code = NotFound desc = could not find container \"4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5\": container with ID starting with 4ec79eda2435b87c9c77762b8eff16d557d7f5a6f6d27b694bb795e078fb65d5 not found: ID does not exist" Sep 17 18:32:08.968075 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968034 2568 scope.go:117] "RemoveContainer" containerID="916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63" Sep 17 18:32:08.968246 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968228 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63"} err="failed to get container status \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": rpc error: code = NotFound desc = could not find container \"916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63\": container with ID starting with 916ad176bb2ea497d62f5c155435d3dd79e531f1ab4def6161945e04eb158e63 not found: ID does not exist" Sep 17 18:32:08.968296 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968245 2568 scope.go:117] "RemoveContainer" containerID="0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00" Sep 17 18:32:08.968407 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968393 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00"} err="failed to get container status \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": rpc error: code = NotFound desc = could not find container \"0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00\": container with ID starting with 0f9497a13623f9a99c3ae79f5de25a992ad45fa13a60075e2231d3f7ac93ee00 not found: ID does not exist" Sep 17 18:32:08.968463 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968407 2568 scope.go:117] "RemoveContainer" containerID="5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e" Sep 17 18:32:08.968651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968636 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e"} err="failed to get container status \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": rpc error: code = NotFound desc = could not find container \"5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e\": container with ID starting with 5b18821a148d7eae56ba20bff495e17847645ea3c71406af6eddd79f60cb245e not found: ID does not exist" Sep 17 18:32:08.968651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968650 2568 scope.go:117] "RemoveContainer" containerID="2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f" Sep 17 18:32:08.968870 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.968851 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f"} err="failed to get container status \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": rpc error: code = NotFound desc = could not find container \"2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f\": container with ID starting with 2364dc1394b5a9fd5214f91d11d904b91104ff0f8fb74b468bf02006cb4f886f not found: ID does not exist" Sep 17 18:32:08.971890 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.971875 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:08.975313 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975292 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 18:32:08.975411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975314 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 18:32:08.975411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975320 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 18:32:08.975411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975332 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 18:32:08.975411 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975293 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ll2j8mrptpea\"" Sep 17 18:32:08.975743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975729 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 18:32:08.975743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975737 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 18:32:08.975890 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975875 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nlfrz\"" Sep 17 18:32:08.975985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.975968 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 18:32:08.976336 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.976311 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 18:32:08.976420 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.976367 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 18:32:08.976491 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.976441 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 18:32:08.976491 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.976480 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 18:32:08.976856 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.976836 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 18:32:08.976944 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.976836 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 18:32:08.981516 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.981497 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 18:32:08.984954 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.984933 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70848e8-df6f-4708-afc5-143655dcc679" path="/var/lib/kubelet/pods/f70848e8-df6f-4708-afc5-143655dcc679/volumes" Sep 17 18:32:08.985857 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.985840 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 18:32:08.993688 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:08.993667 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:32:09.130437 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c872b5-842d-4bd9-8f85-4a6014d169f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130627 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130627 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130521 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130627 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130784 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-config\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130784 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130784 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130705 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130784 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130945 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130945 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130879 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130945 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c872b5-842d-4bd9-8f85-4a6014d169f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.130945 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130931 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131135 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131135 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.130965 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131135 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.131020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht62p\" (UniqueName: \"kubernetes.io/projected/93c872b5-842d-4bd9-8f85-4a6014d169f8-kube-api-access-ht62p\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131135 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.131061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131263 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.131134 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131263 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.131168 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131263 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.131197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.131263 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.131220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.231951 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.231900 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht62p\" (UniqueName: \"kubernetes.io/projected/93c872b5-842d-4bd9-8f85-4a6014d169f8-kube-api-access-ht62p\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.231951 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.231956 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232208 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.231981 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232208 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.231997 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232208 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232208 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232208 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c872b5-842d-4bd9-8f85-4a6014d169f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232217 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-config\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c872b5-842d-4bd9-8f85-4a6014d169f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232753 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232929 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.232985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.232977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.233334 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.233268 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.233422 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.233398 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.235792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.235217 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c872b5-842d-4bd9-8f85-4a6014d169f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.235792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.235347 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.235792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.235548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.235792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.235591 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c872b5-842d-4bd9-8f85-4a6014d169f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.235792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.235667 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.235792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.235688 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.235792 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.235714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.236250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.236227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-config\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.236987 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.236958 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.237101 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.237078 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93c872b5-842d-4bd9-8f85-4a6014d169f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.237941 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.237912 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.238058 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.238035 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.238339 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.238318 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.238384 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.238345 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93c872b5-842d-4bd9-8f85-4a6014d169f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.240239 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.240221 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht62p\" (UniqueName: \"kubernetes.io/projected/93c872b5-842d-4bd9-8f85-4a6014d169f8-kube-api-access-ht62p\") pod \"prometheus-k8s-0\" (UID: \"93c872b5-842d-4bd9-8f85-4a6014d169f8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.283404 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.283363 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:09.569537 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.569512 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:32:09.571143 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:32:09.571113 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c872b5_842d_4bd9_8f85_4a6014d169f8.slice/crio-45aea90f8518386b3c535f3665a0e8b38c64f26acefa41184361c25743bbab05 WatchSource:0}: Error finding container 45aea90f8518386b3c535f3665a0e8b38c64f26acefa41184361c25743bbab05: Status 404 returned error can't find the container with id 45aea90f8518386b3c535f3665a0e8b38c64f26acefa41184361c25743bbab05 Sep 17 18:32:09.907184 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.907149 2568 generic.go:356] "Generic (PLEG): container finished" podID="93c872b5-842d-4bd9-8f85-4a6014d169f8" containerID="dc52824cf56c86c7f62d8b6547bdcc808c3047cc7575b41a75c560fe39fd7f35" exitCode=0 Sep 17 18:32:09.907351 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.907205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerDied","Data":"dc52824cf56c86c7f62d8b6547bdcc808c3047cc7575b41a75c560fe39fd7f35"} Sep 17 18:32:09.907351 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:09.907265 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerStarted","Data":"45aea90f8518386b3c535f3665a0e8b38c64f26acefa41184361c25743bbab05"} Sep 17 18:32:10.915579 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:10.915544 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerStarted","Data":"4d5279ffe89ccdd09a69abe5156303496d6342a03a509e41fae4bfc91780b7c7"} Sep 17 18:32:10.915952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:10.915640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerStarted","Data":"a8f6a77c6ab00a201a957c493b407b8ed26e6d6587e5aa01a322de2900cd43f9"} Sep 17 18:32:10.915952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:10.915676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerStarted","Data":"50f3e29afd28be206983eab684adc4e7fb684138b389f14f0b0c48264c474d36"} Sep 17 18:32:10.915952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:10.915704 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerStarted","Data":"7700a22893687fbf7d9560dc210563fef85a268b1bf350b216817cabe204d87d"} Sep 17 18:32:10.915952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:10.915729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerStarted","Data":"14ed492a73a3a54eb960b33ce036ade1ca116d368128cfd404d994a1ced2665c"} Sep 17 18:32:10.915952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:10.915775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93c872b5-842d-4bd9-8f85-4a6014d169f8","Type":"ContainerStarted","Data":"646934f91513580f14aa5de65679841a42015f117ec1b2733b11637fed95eb83"} Sep 17 18:32:10.954870 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:10.954815 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.931907991 podStartE2EDuration="2.954797413s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.889422ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:10.951879036 +0000 UTC m=+178.231530411" watchObservedRunningTime="2026-09-17 18:32:10.954797413 +0000 UTC m=+178.234448787" Sep 17 18:32:14.284296 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:14.284251 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:32:22.166076 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:22.166042 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c4c575c9-nwzxq"] Sep 17 18:32:47.188499 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.188416 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78c4c575c9-nwzxq" podUID="715c5d7b-d406-4def-bbe4-7e6ea1b95afd" containerName="console" containerID="cri-o://e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3" gracePeriod=15 Sep 17 18:32:47.438786 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.438716 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:32:47.575115 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575077 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-service-ca\") pod \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " Sep 17 18:32:47.575310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575123 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n6pl\" (UniqueName: \"kubernetes.io/projected/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-kube-api-access-4n6pl\") pod \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " Sep 17 18:32:47.575310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575147 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-oauth-serving-cert\") pod \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " Sep 17 18:32:47.575310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575193 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-oauth-config\") pod \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " Sep 17 18:32:47.575310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575212 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-trusted-ca-bundle\") pod \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " Sep 17 18:32:47.575310 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575254 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-serving-cert\") pod \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " Sep 17 18:32:47.575617 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575314 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-config\") pod \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\" (UID: \"715c5d7b-d406-4def-bbe4-7e6ea1b95afd\") " Sep 17 18:32:47.575671 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575613 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-service-ca" (OuterVolumeSpecName: "service-ca") pod "715c5d7b-d406-4def-bbe4-7e6ea1b95afd" (UID: "715c5d7b-d406-4def-bbe4-7e6ea1b95afd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:47.575778 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575747 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "715c5d7b-d406-4def-bbe4-7e6ea1b95afd" (UID: "715c5d7b-d406-4def-bbe4-7e6ea1b95afd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:47.575778 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575671 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "715c5d7b-d406-4def-bbe4-7e6ea1b95afd" (UID: "715c5d7b-d406-4def-bbe4-7e6ea1b95afd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:47.575974 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.575953 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-config" (OuterVolumeSpecName: "console-config") pod "715c5d7b-d406-4def-bbe4-7e6ea1b95afd" (UID: "715c5d7b-d406-4def-bbe4-7e6ea1b95afd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:32:47.577596 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.577561 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "715c5d7b-d406-4def-bbe4-7e6ea1b95afd" (UID: "715c5d7b-d406-4def-bbe4-7e6ea1b95afd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:47.577692 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.577625 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-kube-api-access-4n6pl" (OuterVolumeSpecName: "kube-api-access-4n6pl") pod "715c5d7b-d406-4def-bbe4-7e6ea1b95afd" (UID: "715c5d7b-d406-4def-bbe4-7e6ea1b95afd"). InnerVolumeSpecName "kube-api-access-4n6pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:32:47.577692 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.577664 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "715c5d7b-d406-4def-bbe4-7e6ea1b95afd" (UID: "715c5d7b-d406-4def-bbe4-7e6ea1b95afd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:32:47.676316 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.676279 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-oauth-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:47.676316 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.676309 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-oauth-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:47.676316 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.676318 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-trusted-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:47.676316 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.676328 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:47.676595 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.676336 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-console-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:47.676595 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.676345 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-service-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:47.676595 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:47.676353 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6pl\" (UniqueName: \"kubernetes.io/projected/715c5d7b-d406-4def-bbe4-7e6ea1b95afd-kube-api-access-4n6pl\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:32:48.071603 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.071562 2568 generic.go:356] "Generic (PLEG): container finished" podID="715c5d7b-d406-4def-bbe4-7e6ea1b95afd" containerID="e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3" exitCode=0 Sep 17 18:32:48.071776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.071610 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c4c575c9-nwzxq" Sep 17 18:32:48.071776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.071621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c4c575c9-nwzxq" event={"ID":"715c5d7b-d406-4def-bbe4-7e6ea1b95afd","Type":"ContainerDied","Data":"e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3"} Sep 17 18:32:48.071776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.071668 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c4c575c9-nwzxq" event={"ID":"715c5d7b-d406-4def-bbe4-7e6ea1b95afd","Type":"ContainerDied","Data":"fa06f393b2892a640ff1c680c2cb3deab2ea39990bed774795cf19c006b61613"} Sep 17 18:32:48.071776 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.071696 2568 scope.go:117] "RemoveContainer" containerID="e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3" Sep 17 18:32:48.080925 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.080906 2568 scope.go:117] "RemoveContainer" containerID="e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3" Sep 17 18:32:48.081194 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:32:48.081174 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3\": container with ID starting with e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3 not found: ID does not exist" containerID="e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3" Sep 17 18:32:48.081237 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.081204 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3"} err="failed to get container status \"e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3\": rpc error: code = NotFound desc = could not find container \"e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3\": container with ID starting with e9e68f53134182c7e5397999ab204e2e2b28eaeee30ece18f8e63c1050c5e7b3 not found: ID does not exist" Sep 17 18:32:48.095811 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.095774 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c4c575c9-nwzxq"] Sep 17 18:32:48.100422 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.100393 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78c4c575c9-nwzxq"] Sep 17 18:32:48.984246 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:32:48.984214 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="715c5d7b-d406-4def-bbe4-7e6ea1b95afd" path="/var/lib/kubelet/pods/715c5d7b-d406-4def-bbe4-7e6ea1b95afd/volumes" Sep 17 18:33:09.284435 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:33:09.284397 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:33:09.299965 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:33:09.299933 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:33:10.178041 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:33:10.178016 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:34:14.858905 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:34:14.858877 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:34:14.860740 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:34:14.860713 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:34:14.864406 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:34:14.864387 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:34:14.866280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:34:14.866263 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:34:14.869850 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:34:14.869829 2568 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 18:38:05.380441 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.380365 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-8gkz5"] Sep 17 18:38:05.380841 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.380701 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715c5d7b-d406-4def-bbe4-7e6ea1b95afd" containerName="console" Sep 17 18:38:05.380841 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.380714 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="715c5d7b-d406-4def-bbe4-7e6ea1b95afd" containerName="console" Sep 17 18:38:05.380841 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.380770 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="715c5d7b-d406-4def-bbe4-7e6ea1b95afd" containerName="console" Sep 17 18:38:05.383724 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.383706 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.386114 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.386094 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 17 18:38:05.386219 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.386101 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 17 18:38:05.387165 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.387150 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xblxv\"" Sep 17 18:38:05.387237 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.387196 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 18:38:05.387237 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.387210 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 18:38:05.392493 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.392471 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8gkz5"] Sep 17 18:38:05.477196 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.477171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88fj\" (UniqueName: \"kubernetes.io/projected/092811a0-fae6-4ab6-a753-1f1dc6307337-kube-api-access-f88fj\") pod \"s3-init-8gkz5\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.477335 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.477235 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092811a0-fae6-4ab6-a753-1f1dc6307337-model-cache\") pod \"s3-init-8gkz5\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.578356 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.578327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092811a0-fae6-4ab6-a753-1f1dc6307337-model-cache\") pod \"s3-init-8gkz5\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.578481 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.578372 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88fj\" (UniqueName: \"kubernetes.io/projected/092811a0-fae6-4ab6-a753-1f1dc6307337-kube-api-access-f88fj\") pod \"s3-init-8gkz5\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.578720 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.578696 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092811a0-fae6-4ab6-a753-1f1dc6307337-model-cache\") pod \"s3-init-8gkz5\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.588154 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.588124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88fj\" (UniqueName: \"kubernetes.io/projected/092811a0-fae6-4ab6-a753-1f1dc6307337-kube-api-access-f88fj\") pod \"s3-init-8gkz5\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.698706 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.698650 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8gkz5" Sep 17 18:38:05.973137 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.973051 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8gkz5"] Sep 17 18:38:05.975831 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:38:05.975794 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod092811a0_fae6_4ab6_a753_1f1dc6307337.slice/crio-a3e86ab6f1a5989595e28db04726e6b34ae6513c78be190ee70a286c532f7e42 WatchSource:0}: Error finding container a3e86ab6f1a5989595e28db04726e6b34ae6513c78be190ee70a286c532f7e42: Status 404 returned error can't find the container with id a3e86ab6f1a5989595e28db04726e6b34ae6513c78be190ee70a286c532f7e42 Sep 17 18:38:05.977795 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:05.977780 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:38:06.337906 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:06.337824 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8gkz5" event={"ID":"092811a0-fae6-4ab6-a753-1f1dc6307337","Type":"ContainerStarted","Data":"a3e86ab6f1a5989595e28db04726e6b34ae6513c78be190ee70a286c532f7e42"} Sep 17 18:38:09.353562 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:09.353518 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8gkz5" event={"ID":"092811a0-fae6-4ab6-a753-1f1dc6307337","Type":"ContainerStarted","Data":"d7424ac0d9c21b99eaa9bfc0f68b17204c5937907c9ebec098376854d49da84f"} Sep 17 18:38:11.363205 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:11.363132 2568 generic.go:356] "Generic (PLEG): container finished" podID="092811a0-fae6-4ab6-a753-1f1dc6307337" containerID="d7424ac0d9c21b99eaa9bfc0f68b17204c5937907c9ebec098376854d49da84f" exitCode=0 Sep 17 18:38:11.363545 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:11.363207 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8gkz5" event={"ID":"092811a0-fae6-4ab6-a753-1f1dc6307337","Type":"ContainerDied","Data":"d7424ac0d9c21b99eaa9bfc0f68b17204c5937907c9ebec098376854d49da84f"} Sep 17 18:38:15.581250 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:15.581228 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 17 18:38:16.391697 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:16.391658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8gkz5" event={"ID":"092811a0-fae6-4ab6-a753-1f1dc6307337","Type":"ContainerStarted","Data":"9bbe1b4b3d0abef735d483c73e099bec4f2ad8de3a21b0cd87c56341ae41b57a"} Sep 17 18:38:16.417284 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:38:16.417227 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-8gkz5" podStartSLOduration=2.335885019 podStartE2EDuration="11.417211049s" totalImagesPullingTime="7.380588257s" totalInitContainerRuntime="1.700737773s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:16.415613519 +0000 UTC m=+543.695264894" watchObservedRunningTime="2026-09-17 18:38:16.417211049 +0000 UTC m=+543.696862420" Sep 17 18:39:14.888121 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:14.888092 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:39:14.888780 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:14.888602 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:39:14.894432 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:14.894413 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:39:14.894626 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:14.894603 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:39:20.651224 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:20.651187 2568 generic.go:356] "Generic (PLEG): container finished" podID="092811a0-fae6-4ab6-a753-1f1dc6307337" containerID="9bbe1b4b3d0abef735d483c73e099bec4f2ad8de3a21b0cd87c56341ae41b57a" exitCode=0 Sep 17 18:39:20.651639 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:20.651253 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8gkz5" event={"ID":"092811a0-fae6-4ab6-a753-1f1dc6307337","Type":"ContainerDied","Data":"9bbe1b4b3d0abef735d483c73e099bec4f2ad8de3a21b0cd87c56341ae41b57a"} Sep 17 18:39:21.775354 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:21.775329 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8gkz5" Sep 17 18:39:21.911343 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:21.911259 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092811a0-fae6-4ab6-a753-1f1dc6307337-model-cache\") pod \"092811a0-fae6-4ab6-a753-1f1dc6307337\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " Sep 17 18:39:21.911343 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:21.911307 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f88fj\" (UniqueName: \"kubernetes.io/projected/092811a0-fae6-4ab6-a753-1f1dc6307337-kube-api-access-f88fj\") pod \"092811a0-fae6-4ab6-a753-1f1dc6307337\" (UID: \"092811a0-fae6-4ab6-a753-1f1dc6307337\") " Sep 17 18:39:21.913431 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:21.913406 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092811a0-fae6-4ab6-a753-1f1dc6307337-kube-api-access-f88fj" (OuterVolumeSpecName: "kube-api-access-f88fj") pod "092811a0-fae6-4ab6-a753-1f1dc6307337" (UID: "092811a0-fae6-4ab6-a753-1f1dc6307337"). InnerVolumeSpecName "kube-api-access-f88fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:39:21.933986 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:21.933960 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092811a0-fae6-4ab6-a753-1f1dc6307337-model-cache" (OuterVolumeSpecName: "model-cache") pod "092811a0-fae6-4ab6-a753-1f1dc6307337" (UID: "092811a0-fae6-4ab6-a753-1f1dc6307337"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:39:22.012286 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:22.012257 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092811a0-fae6-4ab6-a753-1f1dc6307337-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:39:22.012286 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:22.012280 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f88fj\" (UniqueName: \"kubernetes.io/projected/092811a0-fae6-4ab6-a753-1f1dc6307337-kube-api-access-f88fj\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:39:22.659931 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:22.659899 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8gkz5" event={"ID":"092811a0-fae6-4ab6-a753-1f1dc6307337","Type":"ContainerDied","Data":"a3e86ab6f1a5989595e28db04726e6b34ae6513c78be190ee70a286c532f7e42"} Sep 17 18:39:22.659931 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:22.659929 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8gkz5" Sep 17 18:39:22.659931 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:22.659937 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e86ab6f1a5989595e28db04726e6b34ae6513c78be190ee70a286c532f7e42" Sep 17 18:39:35.849112 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.849041 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d"] Sep 17 18:39:35.849475 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.849408 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092811a0-fae6-4ab6-a753-1f1dc6307337" containerName="download-hf-model" Sep 17 18:39:35.849475 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.849420 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="092811a0-fae6-4ab6-a753-1f1dc6307337" containerName="download-hf-model" Sep 17 18:39:35.849475 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.849432 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092811a0-fae6-4ab6-a753-1f1dc6307337" containerName="s3-init" Sep 17 18:39:35.849475 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.849437 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="092811a0-fae6-4ab6-a753-1f1dc6307337" containerName="s3-init" Sep 17 18:39:35.849623 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.849526 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="092811a0-fae6-4ab6-a753-1f1dc6307337" containerName="s3-init" Sep 17 18:39:35.852734 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.852719 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:35.855382 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.855358 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Sep 17 18:39:35.856406 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.856388 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 18:39:35.856571 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.856553 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Sep 17 18:39:35.856688 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.856668 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 18:39:35.856767 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.856751 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 18:39:35.856823 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.856778 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\"" Sep 17 18:39:35.856951 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.856938 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 18:39:35.870271 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.870251 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d"] Sep 17 18:39:35.907419 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.907392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:35.907566 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.907423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5bf314-082d-4b5e-a585-09ccce76107d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:35.907566 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.907463 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:35.907566 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.907537 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5bf314-082d-4b5e-a585-09ccce76107d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:35.907717 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.907575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv5sg\" (UniqueName: \"kubernetes.io/projected/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-api-access-dv5sg\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:35.948460 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.948423 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7"] Sep 17 18:39:35.951981 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.951964 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:35.954493 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.954473 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Sep 17 18:39:35.964549 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:35.964526 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7"] Sep 17 18:39:36.008912 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.008884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n229t\" (UniqueName: \"kubernetes.io/projected/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-api-access-n229t\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.009079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.008946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.009079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.008979 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.009079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.008996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.009079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.009315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009089 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.009315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5bf314-082d-4b5e-a585-09ccce76107d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.009315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.009315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5bf314-082d-4b5e-a585-09ccce76107d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.009315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv5sg\" (UniqueName: \"kubernetes.io/projected/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-api-access-dv5sg\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.009856 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009835 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5bf314-082d-4b5e-a585-09ccce76107d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.009910 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009876 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.009910 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.009891 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.011539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.011521 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5bf314-082d-4b5e-a585-09ccce76107d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.018612 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.018583 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv5sg\" (UniqueName: \"kubernetes.io/projected/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-api-access-dv5sg\") pod \"isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.110817 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.110730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.110817 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.110789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.110817 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.110813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.111036 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.110858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.111036 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:39:36.110925 2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 17 18:39:36.111036 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.110986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n229t\" (UniqueName: \"kubernetes.io/projected/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-api-access-n229t\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.111163 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:39:36.111086 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls podName:ae8e5f28-8e80-417b-a62a-935dffbd1bb9 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:36.611054839 +0000 UTC m=+623.890706209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls") pod "isvc-canary-rollback-predictor-7679bbb844-77cd7" (UID: "ae8e5f28-8e80-417b-a62a-935dffbd1bb9") : secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 17 18:39:36.111261 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.111221 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.111482 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.111437 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.111557 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.111541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.120084 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.120065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n229t\" (UniqueName: \"kubernetes.io/projected/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-api-access-n229t\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.162983 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.162953 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:36.444469 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.444429 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d"] Sep 17 18:39:36.446616 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:39:36.446583 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5bf314_082d_4b5e_a585_09ccce76107d.slice/crio-1212df9d35bd446146a0d5aa42f141c0a654fe8bb115f5d26396d99037108b2b WatchSource:0}: Error finding container 1212df9d35bd446146a0d5aa42f141c0a654fe8bb115f5d26396d99037108b2b: Status 404 returned error can't find the container with id 1212df9d35bd446146a0d5aa42f141c0a654fe8bb115f5d26396d99037108b2b Sep 17 18:39:36.615009 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.614972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.617581 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.617548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-77cd7\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:36.722391 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.722318 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerStarted","Data":"b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074"} Sep 17 18:39:36.722391 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.722372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerStarted","Data":"1212df9d35bd446146a0d5aa42f141c0a654fe8bb115f5d26396d99037108b2b"} Sep 17 18:39:36.862180 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:36.862145 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:37.144810 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:37.144646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7"] Sep 17 18:39:37.147080 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:39:37.147048 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8e5f28_8e80_417b_a62a_935dffbd1bb9.slice/crio-653f96e05734abfabc4aa86ba39bae25c3f1da4ea0c81296af86b316997a12cc WatchSource:0}: Error finding container 653f96e05734abfabc4aa86ba39bae25c3f1da4ea0c81296af86b316997a12cc: Status 404 returned error can't find the container with id 653f96e05734abfabc4aa86ba39bae25c3f1da4ea0c81296af86b316997a12cc Sep 17 18:39:37.729201 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:37.729154 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerStarted","Data":"fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3"} Sep 17 18:39:37.729365 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:37.729232 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerStarted","Data":"653f96e05734abfabc4aa86ba39bae25c3f1da4ea0c81296af86b316997a12cc"} Sep 17 18:39:40.743350 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:40.743318 2568 generic.go:356] "Generic (PLEG): container finished" podID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerID="b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074" exitCode=0 Sep 17 18:39:40.743736 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:40.743397 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerDied","Data":"b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074"} Sep 17 18:39:41.752739 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:41.752700 2568 generic.go:356] "Generic (PLEG): container finished" podID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerID="fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3" exitCode=0 Sep 17 18:39:41.753201 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:41.752757 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerDied","Data":"fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3"} Sep 17 18:39:56.857457 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:56.857342 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerStarted","Data":"17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6"} Sep 17 18:39:56.859529 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:56.859497 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerStarted","Data":"63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837"} Sep 17 18:39:59.878743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.878706 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerStarted","Data":"01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5"} Sep 17 18:39:59.879160 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.878807 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:39:59.886611 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.886583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerStarted","Data":"4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7"} Sep 17 18:39:59.886832 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.886802 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:59.886919 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.886867 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:39:59.888325 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.888288 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:39:59.906101 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.906060 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podStartSLOduration=3.905301754 podStartE2EDuration="24.906049219s" totalImagesPullingTime="17.142081891s" totalInitContainerRuntime="3.858665574s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:59.903940789 +0000 UTC m=+647.183592165" watchObservedRunningTime="2026-09-17 18:39:59.906049219 +0000 UTC m=+647.185700593" Sep 17 18:39:59.924915 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:39:59.924878 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podStartSLOduration=3.095312546 podStartE2EDuration="24.924865919s" totalImagesPullingTime="18.158329571s" totalInitContainerRuntime="3.671223802s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:59.924195846 +0000 UTC m=+647.203847233" watchObservedRunningTime="2026-09-17 18:39:59.924865919 +0000 UTC m=+647.204517293" Sep 17 18:40:00.890685 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:00.890637 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:40:00.890685 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:00.890676 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:40:00.891881 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:00.891850 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:40:01.895335 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:01.895301 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:40:05.895334 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:05.895303 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:40:05.895963 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:05.895932 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:40:06.899579 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:06.899552 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:40:06.900021 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:06.899993 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:40:15.896406 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:15.896369 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:40:16.900155 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:16.900107 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:40:25.896811 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:25.896771 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:40:26.900907 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:26.900862 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:40:35.895996 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:35.895955 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:40:36.900813 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:36.900775 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:40:45.896689 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:45.896647 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:40:46.900083 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:46.900046 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:40:55.896649 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:55.896607 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:40:56.900426 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:40:56.900346 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:41:05.896500 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:05.896463 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:41:06.900732 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:06.900691 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Sep 17 18:41:15.897303 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:15.897267 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:41:16.075112 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:16.075075 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7"] Sep 17 18:41:16.075339 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:16.075308 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" containerID="cri-o://63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837" gracePeriod=30 Sep 17 18:41:16.075641 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:16.075394 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kube-rbac-proxy" containerID="cri-o://01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5" gracePeriod=30 Sep 17 18:41:16.076394 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:16.076376 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:41:16.214387 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:16.214358 2568 generic.go:356] "Generic (PLEG): container finished" podID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerID="01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5" exitCode=2 Sep 17 18:41:16.214573 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:16.214427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerDied","Data":"01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5"} Sep 17 18:41:16.895962 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:16.895914 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused" Sep 17 18:41:20.513828 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.513804 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:41:20.616271 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616239 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n229t\" (UniqueName: \"kubernetes.io/projected/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-api-access-n229t\") pod \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " Sep 17 18:41:20.616476 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616278 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls\") pod \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " Sep 17 18:41:20.616476 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616324 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kserve-provision-location\") pod \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " Sep 17 18:41:20.616476 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616340 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-cabundle-cert\") pod \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " Sep 17 18:41:20.616476 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616365 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-rbac-proxy-sar-config\") pod \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\" (UID: \"ae8e5f28-8e80-417b-a62a-935dffbd1bb9\") " Sep 17 18:41:20.616759 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616724 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae8e5f28-8e80-417b-a62a-935dffbd1bb9" (UID: "ae8e5f28-8e80-417b-a62a-935dffbd1bb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:41:20.616759 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616738 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae8e5f28-8e80-417b-a62a-935dffbd1bb9" (UID: "ae8e5f28-8e80-417b-a62a-935dffbd1bb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:41:20.616881 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.616786 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae8e5f28-8e80-417b-a62a-935dffbd1bb9" (UID: "ae8e5f28-8e80-417b-a62a-935dffbd1bb9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:41:20.618538 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.618486 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae8e5f28-8e80-417b-a62a-935dffbd1bb9" (UID: "ae8e5f28-8e80-417b-a62a-935dffbd1bb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:41:20.618636 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.618567 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-api-access-n229t" (OuterVolumeSpecName: "kube-api-access-n229t") pod "ae8e5f28-8e80-417b-a62a-935dffbd1bb9" (UID: "ae8e5f28-8e80-417b-a62a-935dffbd1bb9"). InnerVolumeSpecName "kube-api-access-n229t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:41:20.717758 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.717727 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n229t\" (UniqueName: \"kubernetes.io/projected/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-api-access-n229t\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:20.717758 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.717753 2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-proxy-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:20.717758 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.717763 2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:20.717953 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.717771 2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-cabundle-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:20.717953 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:20.717780 2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8e5f28-8e80-417b-a62a-935dffbd1bb9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:21.237176 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.237134 2568 generic.go:356] "Generic (PLEG): container finished" podID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerID="63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837" exitCode=0 Sep 17 18:41:21.237332 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.237191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerDied","Data":"63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837"} Sep 17 18:41:21.237332 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.237213 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" Sep 17 18:41:21.237332 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.237283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7" event={"ID":"ae8e5f28-8e80-417b-a62a-935dffbd1bb9","Type":"ContainerDied","Data":"653f96e05734abfabc4aa86ba39bae25c3f1da4ea0c81296af86b316997a12cc"} Sep 17 18:41:21.237332 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.237322 2568 scope.go:117] "RemoveContainer" containerID="01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5" Sep 17 18:41:21.245085 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.245062 2568 scope.go:117] "RemoveContainer" containerID="63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837" Sep 17 18:41:21.251906 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.251891 2568 scope.go:117] "RemoveContainer" containerID="fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3" Sep 17 18:41:21.258922 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.258904 2568 scope.go:117] "RemoveContainer" containerID="01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5" Sep 17 18:41:21.259184 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:21.259167 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5\": container with ID starting with 01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5 not found: ID does not exist" containerID="01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5" Sep 17 18:41:21.259233 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.259192 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5"} err="failed to get container status \"01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5\": rpc error: code = NotFound desc = could not find container \"01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5\": container with ID starting with 01dc6528ab8f7e794c02005918e9e9dc5fb741b6bfe92f006975e70aa87620b5 not found: ID does not exist" Sep 17 18:41:21.259233 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.259206 2568 scope.go:117] "RemoveContainer" containerID="63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837" Sep 17 18:41:21.259511 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:21.259487 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837\": container with ID starting with 63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837 not found: ID does not exist" containerID="63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837" Sep 17 18:41:21.259553 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.259521 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837"} err="failed to get container status \"63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837\": rpc error: code = NotFound desc = could not find container \"63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837\": container with ID starting with 63f0a5cc8bbbd179fc9542f1f94afcd31eaef944a5abc76b8c85f64be536e837 not found: ID does not exist" Sep 17 18:41:21.259553 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.259545 2568 scope.go:117] "RemoveContainer" containerID="fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3" Sep 17 18:41:21.259825 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:21.259805 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3\": container with ID starting with fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3 not found: ID does not exist" containerID="fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3" Sep 17 18:41:21.259866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.259835 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3"} err="failed to get container status \"fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3\": rpc error: code = NotFound desc = could not find container \"fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3\": container with ID starting with fe0919fee7725bb026da412edc161a972e38d90a1dacde6316b408c77923f3f3 not found: ID does not exist" Sep 17 18:41:21.272252 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.272232 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7"] Sep 17 18:41:21.282806 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:21.282784 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-77cd7"] Sep 17 18:41:22.983940 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:22.983909 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" path="/var/lib/kubelet/pods/ae8e5f28-8e80-417b-a62a-935dffbd1bb9/volumes" Sep 17 18:41:25.600965 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:25.600933 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d"] Sep 17 18:41:25.601359 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:25.601164 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" containerID="cri-o://17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6" gracePeriod=30 Sep 17 18:41:25.601359 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:25.601210 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kube-rbac-proxy" containerID="cri-o://4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7" gracePeriod=30 Sep 17 18:41:25.891432 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:25.891399 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 17 18:41:25.896562 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:25.896540 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 17 18:41:26.264123 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:26.264043 2568 generic.go:356] "Generic (PLEG): container finished" podID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerID="4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7" exitCode=2 Sep 17 18:41:26.264266 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:26.264114 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerDied","Data":"4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7"} Sep 17 18:41:30.143116 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.143090 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:41:30.197625 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.197601 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv5sg\" (UniqueName: \"kubernetes.io/projected/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-api-access-dv5sg\") pod \"6e5bf314-082d-4b5e-a585-09ccce76107d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " Sep 17 18:41:30.197771 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.197636 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-cabundle-cert\") pod \"6e5bf314-082d-4b5e-a585-09ccce76107d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " Sep 17 18:41:30.197771 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.197673 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5bf314-082d-4b5e-a585-09ccce76107d-proxy-tls\") pod \"6e5bf314-082d-4b5e-a585-09ccce76107d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " Sep 17 18:41:30.197771 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.197698 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5bf314-082d-4b5e-a585-09ccce76107d-kserve-provision-location\") pod \"6e5bf314-082d-4b5e-a585-09ccce76107d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " Sep 17 18:41:30.197933 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.197813 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-rbac-proxy-sar-config\") pod \"6e5bf314-082d-4b5e-a585-09ccce76107d\" (UID: \"6e5bf314-082d-4b5e-a585-09ccce76107d\") " Sep 17 18:41:30.198122 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.198036 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5bf314-082d-4b5e-a585-09ccce76107d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e5bf314-082d-4b5e-a585-09ccce76107d" (UID: "6e5bf314-082d-4b5e-a585-09ccce76107d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:41:30.198122 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.198105 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e5bf314-082d-4b5e-a585-09ccce76107d" (UID: "6e5bf314-082d-4b5e-a585-09ccce76107d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:41:30.198314 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.198221 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e5bf314-082d-4b5e-a585-09ccce76107d" (UID: "6e5bf314-082d-4b5e-a585-09ccce76107d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:41:30.199671 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.199645 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-api-access-dv5sg" (OuterVolumeSpecName: "kube-api-access-dv5sg") pod "6e5bf314-082d-4b5e-a585-09ccce76107d" (UID: "6e5bf314-082d-4b5e-a585-09ccce76107d"). InnerVolumeSpecName "kube-api-access-dv5sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:41:30.199775 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.199696 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5bf314-082d-4b5e-a585-09ccce76107d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e5bf314-082d-4b5e-a585-09ccce76107d" (UID: "6e5bf314-082d-4b5e-a585-09ccce76107d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:41:30.282237 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.282181 2568 generic.go:356] "Generic (PLEG): container finished" podID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerID="17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6" exitCode=0 Sep 17 18:41:30.282237 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.282236 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" Sep 17 18:41:30.282372 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.282260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerDied","Data":"17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6"} Sep 17 18:41:30.282372 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.282306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d" event={"ID":"6e5bf314-082d-4b5e-a585-09ccce76107d","Type":"ContainerDied","Data":"1212df9d35bd446146a0d5aa42f141c0a654fe8bb115f5d26396d99037108b2b"} Sep 17 18:41:30.282372 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.282331 2568 scope.go:117] "RemoveContainer" containerID="4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7" Sep 17 18:41:30.291299 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.291284 2568 scope.go:117] "RemoveContainer" containerID="17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6" Sep 17 18:41:30.298885 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.298868 2568 scope.go:117] "RemoveContainer" containerID="b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074" Sep 17 18:41:30.298948 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.298933 2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:30.299003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.298952 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv5sg\" (UniqueName: \"kubernetes.io/projected/6e5bf314-082d-4b5e-a585-09ccce76107d-kube-api-access-dv5sg\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:30.299003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.298961 2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e5bf314-082d-4b5e-a585-09ccce76107d-cabundle-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:30.299003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.298969 2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e5bf314-082d-4b5e-a585-09ccce76107d-proxy-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:30.299003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.298982 2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e5bf314-082d-4b5e-a585-09ccce76107d-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:41:30.306181 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.306160 2568 scope.go:117] "RemoveContainer" containerID="4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7" Sep 17 18:41:30.306249 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.306171 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d"] Sep 17 18:41:30.306465 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:30.306426 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7\": container with ID starting with 4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7 not found: ID does not exist" containerID="4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7" Sep 17 18:41:30.306526 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.306470 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7"} err="failed to get container status \"4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7\": rpc error: code = NotFound desc = could not find container \"4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7\": container with ID starting with 4c98479f9ea7ef073a8f8e91b40502b978bba9b93d4dca0cf123b12aa09502d7 not found: ID does not exist" Sep 17 18:41:30.306526 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.306487 2568 scope.go:117] "RemoveContainer" containerID="17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6" Sep 17 18:41:30.306725 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:30.306707 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6\": container with ID starting with 17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6 not found: ID does not exist" containerID="17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6" Sep 17 18:41:30.306808 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.306730 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6"} err="failed to get container status \"17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6\": rpc error: code = NotFound desc = could not find container \"17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6\": container with ID starting with 17e40b69a9c67339740da8235efbb926153a12b05b1e92747bd933b33a2937f6 not found: ID does not exist" Sep 17 18:41:30.306808 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.306742 2568 scope.go:117] "RemoveContainer" containerID="b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074" Sep 17 18:41:30.306989 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:30.306972 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074\": container with ID starting with b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074 not found: ID does not exist" containerID="b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074" Sep 17 18:41:30.307035 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.306994 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074"} err="failed to get container status \"b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074\": rpc error: code = NotFound desc = could not find container \"b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074\": container with ID starting with b352058b5813925b2990cd50a6100628cfdddc8e7306cebbb5439a1022eb8074 not found: ID does not exist" Sep 17 18:41:30.311675 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.311655 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-cf97f779c-n8p7d"] Sep 17 18:41:30.820066 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820030 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6"] Sep 17 18:41:30.820396 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820384 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" Sep 17 18:41:30.820440 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820399 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" Sep 17 18:41:30.820440 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820411 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kube-rbac-proxy" Sep 17 18:41:30.820440 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820417 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kube-rbac-proxy" Sep 17 18:41:30.820440 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820429 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kube-rbac-proxy" Sep 17 18:41:30.820440 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820437 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kube-rbac-proxy" Sep 17 18:41:30.820440 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820461 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="storage-initializer" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820468 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="storage-initializer" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820484 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820489 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820497 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="storage-initializer" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820502 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="storage-initializer" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820559 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kserve-container" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820569 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kube-rbac-proxy" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820575 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" containerName="kserve-container" Sep 17 18:41:30.820643 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.820581 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8e5f28-8e80-417b-a62a-935dffbd1bb9" containerName="kube-rbac-proxy" Sep 17 18:41:30.823936 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.823907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:30.826162 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.826132 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e4246-predictor-serving-cert\"" Sep 17 18:41:30.826773 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.826752 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e4246-kube-rbac-proxy-sar-config\"" Sep 17 18:41:30.826853 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.826783 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 18:41:30.826905 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.826848 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 18:41:30.827030 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.827016 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\"" Sep 17 18:41:30.827208 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.827194 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 18:41:30.827256 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.827240 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 18:41:30.836625 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.836603 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6"] Sep 17 18:41:30.905113 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.905078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:30.905287 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.905124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:30.905287 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.905145 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:30.905287 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.905251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhzk\" (UniqueName: \"kubernetes.io/projected/b1902a41-7131-47f5-b912-3973a044d663-kube-api-access-hhhzk\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:30.905438 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.905321 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1902a41-7131-47f5-b912-3973a044d663-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:30.984024 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:30.983993 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5bf314-082d-4b5e-a585-09ccce76107d" path="/var/lib/kubelet/pods/6e5bf314-082d-4b5e-a585-09ccce76107d/volumes" Sep 17 18:41:31.005869 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.005834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.006027 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.005883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.006027 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.005944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhzk\" (UniqueName: \"kubernetes.io/projected/b1902a41-7131-47f5-b912-3973a044d663-kube-api-access-hhhzk\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.006027 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:31.005992 2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-serving-cert: secret "isvc-sklearn-graph-raw-e4246-predictor-serving-cert" not found Sep 17 18:41:31.006179 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.006028 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1902a41-7131-47f5-b912-3973a044d663-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.006179 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:41:31.006073 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls podName:b1902a41-7131-47f5-b912-3973a044d663 nodeName:}" failed. No retries permitted until 2026-09-17 18:41:31.506050893 +0000 UTC m=+738.785702249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls") pod "isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" (UID: "b1902a41-7131-47f5-b912-3973a044d663") : secret "isvc-sklearn-graph-raw-e4246-predictor-serving-cert" not found Sep 17 18:41:31.006179 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.006143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.006419 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.006400 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1902a41-7131-47f5-b912-3973a044d663-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.006578 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.006561 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.006706 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.006684 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.017164 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.017142 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhzk\" (UniqueName: \"kubernetes.io/projected/b1902a41-7131-47f5-b912-3973a044d663-kube-api-access-hhhzk\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.509745 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.509717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.512025 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.512005 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls\") pod \"isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:31.734556 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:31.734525 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:32.006808 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:32.006780 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6"] Sep 17 18:41:32.008624 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:41:32.008596 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1902a41_7131_47f5_b912_3973a044d663.slice/crio-dcf7e5b20d296458e0a07a939c20704bcb3bf847fb3d8807b5417085671238b9 WatchSource:0}: Error finding container dcf7e5b20d296458e0a07a939c20704bcb3bf847fb3d8807b5417085671238b9: Status 404 returned error can't find the container with id dcf7e5b20d296458e0a07a939c20704bcb3bf847fb3d8807b5417085671238b9 Sep 17 18:41:32.299592 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:32.299505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerStarted","Data":"7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e"} Sep 17 18:41:32.299735 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:32.299607 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerStarted","Data":"dcf7e5b20d296458e0a07a939c20704bcb3bf847fb3d8807b5417085671238b9"} Sep 17 18:41:36.317991 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:36.317909 2568 generic.go:356] "Generic (PLEG): container finished" podID="b1902a41-7131-47f5-b912-3973a044d663" containerID="7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e" exitCode=0 Sep 17 18:41:36.317991 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:36.317980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerDied","Data":"7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e"} Sep 17 18:41:37.324760 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:37.324724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerStarted","Data":"d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682"} Sep 17 18:41:37.325146 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:37.324787 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerStarted","Data":"cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf"} Sep 17 18:41:37.325146 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:37.325050 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:37.348775 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:37.348730 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podStartSLOduration=3.503673813 podStartE2EDuration="7.348718321s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845044508s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:37.346463806 +0000 UTC m=+744.626115173" watchObservedRunningTime="2026-09-17 18:41:37.348718321 +0000 UTC m=+744.628369694" Sep 17 18:41:38.328759 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:38.328720 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:38.329973 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:38.329948 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:41:39.333054 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:39.333012 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:41:44.337492 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:44.337461 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:41:44.338089 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:44.338065 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:41:54.338048 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:41:54.338008 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:04.338408 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:42:04.338368 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:14.338204 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:42:14.338170 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:24.338710 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:42:24.338663 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:34.338472 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:42:34.338367 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:44.338857 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:42:44.338807 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:42:54.339138 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:42:54.339110 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:43:21.039333 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.039298 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6"] Sep 17 18:43:21.039900 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.039555 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" containerID="cri-o://cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf" gracePeriod=30 Sep 17 18:43:21.039900 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.039606 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kube-rbac-proxy" containerID="cri-o://d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682" gracePeriod=30 Sep 17 18:43:21.111871 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.111835 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b"] Sep 17 18:43:21.117089 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.117068 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.119809 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.119784 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7a45d-predictor-serving-cert\"" Sep 17 18:43:21.120083 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.120065 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7a45d-kube-rbac-proxy-sar-config\"" Sep 17 18:43:21.130493 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.130464 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b"] Sep 17 18:43:21.228499 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.228468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.228677 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.228529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.228677 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.228556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.228677 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.228583 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.228677 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.228613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl758\" (UniqueName: \"kubernetes.io/projected/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-api-access-nl758\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330015 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.329916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330015 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.329958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl758\" (UniqueName: \"kubernetes.io/projected/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-api-access-nl758\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330015 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.330011 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330309 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.330039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330309 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.330058 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330309 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:43:21.330163 2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-7a45d-predictor-serving-cert" not found Sep 17 18:43:21.330309 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:43:21.330223 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls podName:a2959bbf-0390-4f9d-8d70-0bd90663fb50 nodeName:}" failed. No retries permitted until 2026-09-17 18:43:21.830204088 +0000 UTC m=+849.109855462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" (UID: "a2959bbf-0390-4f9d-8d70-0bd90663fb50") : secret "isvc-sklearn-graph-raw-hpa-7a45d-predictor-serving-cert" not found Sep 17 18:43:21.330593 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.330438 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330752 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.330731 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.330828 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.330798 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.340561 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.340537 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl758\" (UniqueName: \"kubernetes.io/projected/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-api-access-nl758\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.775008 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.774975 2568 generic.go:356] "Generic (PLEG): container finished" podID="b1902a41-7131-47f5-b912-3973a044d663" containerID="d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682" exitCode=2 Sep 17 18:43:21.775181 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.775047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerDied","Data":"d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682"} Sep 17 18:43:21.834866 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.834829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:21.837527 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:21.837496 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:22.028814 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:22.028730 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:22.310902 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:22.310787 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b"] Sep 17 18:43:22.313244 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:43:22.313216 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2959bbf_0390_4f9d_8d70_0bd90663fb50.slice/crio-292c9ff7f3eb098daa1495e5079102fd385da0f7461aa29f87ca47af1535c4ec WatchSource:0}: Error finding container 292c9ff7f3eb098daa1495e5079102fd385da0f7461aa29f87ca47af1535c4ec: Status 404 returned error can't find the container with id 292c9ff7f3eb098daa1495e5079102fd385da0f7461aa29f87ca47af1535c4ec Sep 17 18:43:22.782003 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:22.781964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerStarted","Data":"f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4"} Sep 17 18:43:22.782173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:22.782023 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerStarted","Data":"292c9ff7f3eb098daa1495e5079102fd385da0f7461aa29f87ca47af1535c4ec"} Sep 17 18:43:24.333872 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:24.333829 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused" Sep 17 18:43:24.338163 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:24.338141 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused" Sep 17 18:43:25.781624 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.781594 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:43:25.796740 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.796708 2568 generic.go:356] "Generic (PLEG): container finished" podID="b1902a41-7131-47f5-b912-3973a044d663" containerID="cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf" exitCode=0 Sep 17 18:43:25.796853 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.796770 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" Sep 17 18:43:25.796853 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.796775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerDied","Data":"cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf"} Sep 17 18:43:25.796965 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.796876 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6" event={"ID":"b1902a41-7131-47f5-b912-3973a044d663","Type":"ContainerDied","Data":"dcf7e5b20d296458e0a07a939c20704bcb3bf847fb3d8807b5417085671238b9"} Sep 17 18:43:25.796965 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.796921 2568 scope.go:117] "RemoveContainer" containerID="d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682" Sep 17 18:43:25.806638 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.806614 2568 scope.go:117] "RemoveContainer" containerID="cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf" Sep 17 18:43:25.815895 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.815869 2568 scope.go:117] "RemoveContainer" containerID="7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e" Sep 17 18:43:25.823313 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.823298 2568 scope.go:117] "RemoveContainer" containerID="d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682" Sep 17 18:43:25.823641 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:43:25.823619 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682\": container with ID starting with d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682 not found: ID does not exist" containerID="d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682" Sep 17 18:43:25.823703 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.823651 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682"} err="failed to get container status \"d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682\": rpc error: code = NotFound desc = could not find container \"d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682\": container with ID starting with d2a66a009a36d977374a366b814dfb05c214d8a7bb25f84f1ad5779c08fbd682 not found: ID does not exist" Sep 17 18:43:25.823703 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.823669 2568 scope.go:117] "RemoveContainer" containerID="cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf" Sep 17 18:43:25.823940 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:43:25.823922 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf\": container with ID starting with cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf not found: ID does not exist" containerID="cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf" Sep 17 18:43:25.823995 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.823945 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf"} err="failed to get container status \"cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf\": rpc error: code = NotFound desc = could not find container \"cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf\": container with ID starting with cb6983ea79f219eeea205c50dcc7dd1be3a0745645cd4292e76b0eb75c4080bf not found: ID does not exist" Sep 17 18:43:25.823995 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.823959 2568 scope.go:117] "RemoveContainer" containerID="7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e" Sep 17 18:43:25.824188 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:43:25.824167 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e\": container with ID starting with 7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e not found: ID does not exist" containerID="7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e" Sep 17 18:43:25.824258 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.824193 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e"} err="failed to get container status \"7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e\": rpc error: code = NotFound desc = could not find container \"7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e\": container with ID starting with 7d457bc984a14ba5a0975703553a839bc4d603285de3550011618a87d0fcee3e not found: ID does not exist" Sep 17 18:43:25.870613 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.870582 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls\") pod \"b1902a41-7131-47f5-b912-3973a044d663\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " Sep 17 18:43:25.870736 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.870685 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhhzk\" (UniqueName: \"kubernetes.io/projected/b1902a41-7131-47f5-b912-3973a044d663-kube-api-access-hhhzk\") pod \"b1902a41-7131-47f5-b912-3973a044d663\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " Sep 17 18:43:25.870736 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.870715 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1902a41-7131-47f5-b912-3973a044d663-kserve-provision-location\") pod \"b1902a41-7131-47f5-b912-3973a044d663\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " Sep 17 18:43:25.870861 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.870740 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-cabundle-cert\") pod \"b1902a41-7131-47f5-b912-3973a044d663\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " Sep 17 18:43:25.870861 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.870772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-kube-rbac-proxy-sar-config\") pod \"b1902a41-7131-47f5-b912-3973a044d663\" (UID: \"b1902a41-7131-47f5-b912-3973a044d663\") " Sep 17 18:43:25.871093 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.871068 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1902a41-7131-47f5-b912-3973a044d663-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1902a41-7131-47f5-b912-3973a044d663" (UID: "b1902a41-7131-47f5-b912-3973a044d663"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:43:25.871174 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.871126 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1902a41-7131-47f5-b912-3973a044d663" (UID: "b1902a41-7131-47f5-b912-3973a044d663"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:43:25.871174 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.871139 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1902a41-7131-47f5-b912-3973a044d663" (UID: "b1902a41-7131-47f5-b912-3973a044d663"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:43:25.872722 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.872700 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1902a41-7131-47f5-b912-3973a044d663" (UID: "b1902a41-7131-47f5-b912-3973a044d663"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:43:25.872838 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.872825 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1902a41-7131-47f5-b912-3973a044d663-kube-api-access-hhhzk" (OuterVolumeSpecName: "kube-api-access-hhhzk") pod "b1902a41-7131-47f5-b912-3973a044d663" (UID: "b1902a41-7131-47f5-b912-3973a044d663"). InnerVolumeSpecName "kube-api-access-hhhzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:43:25.972408 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.972377 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhhzk\" (UniqueName: \"kubernetes.io/projected/b1902a41-7131-47f5-b912-3973a044d663-kube-api-access-hhhzk\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:43:25.972408 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.972405 2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1902a41-7131-47f5-b912-3973a044d663-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:43:25.972581 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.972416 2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-cabundle-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:43:25.972581 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.972425 2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1902a41-7131-47f5-b912-3973a044d663-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:43:25.972581 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:25.972434 2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1902a41-7131-47f5-b912-3973a044d663-proxy-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:43:26.121570 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:26.121538 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6"] Sep 17 18:43:26.128319 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:26.128295 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e4246-predictor-6fb78f4dcc-7ndq6"] Sep 17 18:43:26.807057 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:26.807017 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerID="f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4" exitCode=0 Sep 17 18:43:26.807499 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:26.807088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerDied","Data":"f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4"} Sep 17 18:43:26.984812 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:26.984783 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1902a41-7131-47f5-b912-3973a044d663" path="/var/lib/kubelet/pods/b1902a41-7131-47f5-b912-3973a044d663/volumes" Sep 17 18:43:27.814113 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:27.814076 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerStarted","Data":"a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7"} Sep 17 18:43:27.814512 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:27.814153 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerStarted","Data":"dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da"} Sep 17 18:43:27.814512 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:27.814349 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:27.835779 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:27.835723 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podStartSLOduration=3.241080606 podStartE2EDuration="6.835707396s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.59462679s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:27.834174829 +0000 UTC m=+855.113826201" watchObservedRunningTime="2026-09-17 18:43:27.835707396 +0000 UTC m=+855.115358767" Sep 17 18:43:28.818360 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:28.818327 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:28.819502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:28.819475 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:43:29.827553 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:29.827509 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:43:34.830350 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:34.830319 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:43:34.830849 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:34.830820 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:43:44.830828 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:44.830788 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:43:54.831381 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:43:54.831338 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:44:04.831079 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:04.830994 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:44:14.830793 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:14.830751 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:44:14.917224 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:14.917193 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:44:14.918516 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:14.918496 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:44:14.923277 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:14.923261 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:44:14.924561 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:14.924540 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:44:24.831333 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:24.831294 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:44:34.831603 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:34.831564 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:44:44.831607 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:44:44.831577 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:45:11.379035 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:11.378999 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b"] Sep 17 18:45:11.381677 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:11.379234 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" containerID="cri-o://dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da" gracePeriod=30 Sep 17 18:45:11.381677 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:11.379289 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kube-rbac-proxy" containerID="cri-o://a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7" gracePeriod=30 Sep 17 18:45:12.262297 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:12.262255 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerID="a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7" exitCode=2 Sep 17 18:45:12.262495 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:12.262322 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerDied","Data":"a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7"} Sep 17 18:45:14.826309 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:14.826269 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused" Sep 17 18:45:14.831660 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:14.831633 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Sep 17 18:45:16.024073 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.024045 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:45:16.106656 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.106624 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl758\" (UniqueName: \"kubernetes.io/projected/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-api-access-nl758\") pod \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " Sep 17 18:45:16.106831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.106670 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls\") pod \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " Sep 17 18:45:16.106831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.106690 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-rbac-proxy-sar-config\") pod \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " Sep 17 18:45:16.106831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.106741 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-cabundle-cert\") pod \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " Sep 17 18:45:16.107097 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.107073 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2959bbf-0390-4f9d-8d70-0bd90663fb50" (UID: "a2959bbf-0390-4f9d-8d70-0bd90663fb50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:45:16.107161 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.107076 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2959bbf-0390-4f9d-8d70-0bd90663fb50" (UID: "a2959bbf-0390-4f9d-8d70-0bd90663fb50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:45:16.108823 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.108797 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2959bbf-0390-4f9d-8d70-0bd90663fb50" (UID: "a2959bbf-0390-4f9d-8d70-0bd90663fb50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:45:16.108923 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.108801 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-api-access-nl758" (OuterVolumeSpecName: "kube-api-access-nl758") pod "a2959bbf-0390-4f9d-8d70-0bd90663fb50" (UID: "a2959bbf-0390-4f9d-8d70-0bd90663fb50"). InnerVolumeSpecName "kube-api-access-nl758". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:45:16.207886 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.207812 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kserve-provision-location\") pod \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\" (UID: \"a2959bbf-0390-4f9d-8d70-0bd90663fb50\") " Sep 17 18:45:16.208016 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.207976 2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-cabundle-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:45:16.208016 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.207986 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl758\" (UniqueName: \"kubernetes.io/projected/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-api-access-nl758\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:45:16.208016 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.207995 2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2959bbf-0390-4f9d-8d70-0bd90663fb50-proxy-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:45:16.208016 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.208003 2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:45:16.208147 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.208075 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2959bbf-0390-4f9d-8d70-0bd90663fb50" (UID: "a2959bbf-0390-4f9d-8d70-0bd90663fb50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:16.279222 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.279192 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerID="dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da" exitCode=0 Sep 17 18:45:16.279373 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.279253 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" Sep 17 18:45:16.279373 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.279275 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerDied","Data":"dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da"} Sep 17 18:45:16.279483 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.279387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b" event={"ID":"a2959bbf-0390-4f9d-8d70-0bd90663fb50","Type":"ContainerDied","Data":"292c9ff7f3eb098daa1495e5079102fd385da0f7461aa29f87ca47af1535c4ec"} Sep 17 18:45:16.279483 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.279426 2568 scope.go:117] "RemoveContainer" containerID="a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7" Sep 17 18:45:16.290591 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.290574 2568 scope.go:117] "RemoveContainer" containerID="dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da" Sep 17 18:45:16.297720 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.297702 2568 scope.go:117] "RemoveContainer" containerID="f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4" Sep 17 18:45:16.303952 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.303929 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b"] Sep 17 18:45:16.305208 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.305190 2568 scope.go:117] "RemoveContainer" containerID="a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7" Sep 17 18:45:16.305462 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:45:16.305431 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7\": container with ID starting with a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7 not found: ID does not exist" containerID="a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7" Sep 17 18:45:16.305523 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.305468 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7"} err="failed to get container status \"a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7\": rpc error: code = NotFound desc = could not find container \"a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7\": container with ID starting with a0517a3e312983e168a702154003bf2afefd7ccfc001bc5d249e085223c07ab7 not found: ID does not exist" Sep 17 18:45:16.305523 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.305487 2568 scope.go:117] "RemoveContainer" containerID="dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da" Sep 17 18:45:16.305733 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:45:16.305715 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da\": container with ID starting with dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da not found: ID does not exist" containerID="dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da" Sep 17 18:45:16.305796 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.305743 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da"} err="failed to get container status \"dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da\": rpc error: code = NotFound desc = could not find container \"dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da\": container with ID starting with dadddc62f4cab2eb3626f72d91c5cfe683f46a7cd8596b72855fa52ce7a563da not found: ID does not exist" Sep 17 18:45:16.305796 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.305765 2568 scope.go:117] "RemoveContainer" containerID="f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4" Sep 17 18:45:16.305980 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:45:16.305966 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4\": container with ID starting with f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4 not found: ID does not exist" containerID="f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4" Sep 17 18:45:16.306023 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.305985 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4"} err="failed to get container status \"f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4\": rpc error: code = NotFound desc = could not find container \"f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4\": container with ID starting with f66130b0dd892ca92ed6be130490862eea45fe68f529073c9b97ae46d5290ca4 not found: ID does not exist" Sep 17 18:45:16.308380 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.308366 2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2959bbf-0390-4f9d-8d70-0bd90663fb50-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:45:16.313576 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.313553 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7a45d-predictor-6d497df8d9-xv58b"] Sep 17 18:45:16.984686 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:45:16.984654 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" path="/var/lib/kubelet/pods/a2959bbf-0390-4f9d-8d70-0bd90663fb50/volumes" Sep 17 18:47:06.948984 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.948902 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5"] Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949245 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kube-rbac-proxy" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949258 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kube-rbac-proxy" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949272 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kube-rbac-proxy" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949279 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kube-rbac-proxy" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949287 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="storage-initializer" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949293 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="storage-initializer" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949302 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949307 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949313 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="storage-initializer" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949318 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="storage-initializer" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949324 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949331 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949387 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kube-rbac-proxy" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949399 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kube-rbac-proxy" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949406 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2959bbf-0390-4f9d-8d70-0bd90663fb50" containerName="kserve-container" Sep 17 18:47:06.949506 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.949412 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1902a41-7131-47f5-b912-3973a044d663" containerName="kserve-container" Sep 17 18:47:06.952497 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.952481 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:06.956424 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.956399 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a1be8-kube-rbac-proxy-sar-config\"" Sep 17 18:47:06.956714 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.956690 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 17 18:47:06.956823 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.956802 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a1be8-predictor-serving-cert\"" Sep 17 18:47:06.956898 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.956801 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 18:47:06.957285 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.957269 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\"" Sep 17 18:47:06.957330 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.957295 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 17 18:47:06.957375 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.957272 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 18:47:06.968651 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:06.968629 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5"] Sep 17 18:47:07.072536 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.072501 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.072709 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.072577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.072709 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.072607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-proxy-tls\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.072709 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.072655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.072709 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.072704 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2kl\" (UniqueName: \"kubernetes.io/projected/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-api-access-cz2kl\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.173441 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.173411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.173604 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.173486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.173604 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.173507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-proxy-tls\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.173697 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.173617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.173697 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.173661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2kl\" (UniqueName: \"kubernetes.io/projected/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-api-access-cz2kl\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.174005 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.173981 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.174245 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.174219 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.174326 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.174221 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.175957 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.175936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-proxy-tls\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.183280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.183262 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2kl\" (UniqueName: \"kubernetes.io/projected/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-api-access-cz2kl\") pod \"isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.262277 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.262181 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:07.541033 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.541008 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5"] Sep 17 18:47:07.543577 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:47:07.543549 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cc4ced_dfcd_41d7_b5b8_0492a98b3406.slice/crio-47456381b6ba62600d327ef7a73dc0216653fa27768dd6a615e3777e25b10830 WatchSource:0}: Error finding container 47456381b6ba62600d327ef7a73dc0216653fa27768dd6a615e3777e25b10830: Status 404 returned error can't find the container with id 47456381b6ba62600d327ef7a73dc0216653fa27768dd6a615e3777e25b10830 Sep 17 18:47:07.733030 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.732997 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerStarted","Data":"03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986"} Sep 17 18:47:07.733198 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:07.733059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerStarted","Data":"47456381b6ba62600d327ef7a73dc0216653fa27768dd6a615e3777e25b10830"} Sep 17 18:47:11.749994 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:11.749903 2568 generic.go:356] "Generic (PLEG): container finished" podID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerID="03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986" exitCode=0 Sep 17 18:47:11.750477 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:11.749980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerDied","Data":"03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986"} Sep 17 18:47:12.755884 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:12.755849 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerStarted","Data":"b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a"} Sep 17 18:47:12.756510 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:12.755921 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerStarted","Data":"e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3"} Sep 17 18:47:12.756510 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:12.756135 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:12.789377 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:12.789325 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podStartSLOduration=2.8958931999999997 podStartE2EDuration="6.789309505s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.893416305s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:12.788414615 +0000 UTC m=+1080.068066002" watchObservedRunningTime="2026-09-17 18:47:12.789309505 +0000 UTC m=+1080.068960878" Sep 17 18:47:13.760586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:13.760554 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:13.761879 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:13.761850 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:14.764969 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:14.764931 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:19.769391 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:19.769361 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:47:19.769939 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:19.769912 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:29.769808 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:29.769768 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:39.770051 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:39.770005 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:49.770454 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:49.770402 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:47:59.770281 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:47:59.770240 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:48:09.769968 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:48:09.769924 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:48:19.769813 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:48:19.769770 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:48:29.770342 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:48:29.770253 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:48:37.979632 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:48:37.979581 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:48:47.979587 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:48:47.979546 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:48:57.979610 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:48:57.979562 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:49:07.980532 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:07.980486 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:49:14.947280 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:14.947247 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:49:14.950203 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:14.950179 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:49:14.953363 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:14.953343 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:49:14.956406 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:14.956385 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:49:17.980053 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:17.980004 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:49:27.979641 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:27.979595 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:49:37.980618 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:37.980572 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:49:46.983889 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:46.983862 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:49:56.989831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:56.989754 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5"] Sep 17 18:49:56.990230 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:56.990001 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" containerID="cri-o://e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3" gracePeriod=30 Sep 17 18:49:56.990230 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:56.990078 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kube-rbac-proxy" containerID="cri-o://b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a" gracePeriod=30 Sep 17 18:49:57.445861 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:57.445823 2568 generic.go:356] "Generic (PLEG): container finished" podID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerID="b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a" exitCode=2 Sep 17 18:49:57.446029 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:57.445875 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerDied","Data":"b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a"} Sep 17 18:49:59.765748 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:49:59.765701 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused" Sep 17 18:50:04.765746 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:04.765701 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused" Sep 17 18:50:06.979657 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:06.979620 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Sep 17 18:50:07.128375 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.128350 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:50:07.197082 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197049 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-rbac-proxy-sar-config\") pod \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " Sep 17 18:50:07.197228 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197098 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-cabundle-cert\") pod \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " Sep 17 18:50:07.197228 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197121 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kserve-provision-location\") pod \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " Sep 17 18:50:07.197228 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197146 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-proxy-tls\") pod \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " Sep 17 18:50:07.197228 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197207 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz2kl\" (UniqueName: \"kubernetes.io/projected/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-api-access-cz2kl\") pod \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\" (UID: \"50cc4ced-dfcd-41d7-b5b8-0492a98b3406\") " Sep 17 18:50:07.197490 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197439 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50cc4ced-dfcd-41d7-b5b8-0492a98b3406" (UID: "50cc4ced-dfcd-41d7-b5b8-0492a98b3406"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:50:07.197588 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197571 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50cc4ced-dfcd-41d7-b5b8-0492a98b3406" (UID: "50cc4ced-dfcd-41d7-b5b8-0492a98b3406"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:50:07.197691 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.197661 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50cc4ced-dfcd-41d7-b5b8-0492a98b3406" (UID: "50cc4ced-dfcd-41d7-b5b8-0492a98b3406"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:50:07.199371 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.199349 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-api-access-cz2kl" (OuterVolumeSpecName: "kube-api-access-cz2kl") pod "50cc4ced-dfcd-41d7-b5b8-0492a98b3406" (UID: "50cc4ced-dfcd-41d7-b5b8-0492a98b3406"). InnerVolumeSpecName "kube-api-access-cz2kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:50:07.199371 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.199357 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50cc4ced-dfcd-41d7-b5b8-0492a98b3406" (UID: "50cc4ced-dfcd-41d7-b5b8-0492a98b3406"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:50:07.298747 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.298646 2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-proxy-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:50:07.298747 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.298680 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2kl\" (UniqueName: \"kubernetes.io/projected/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-api-access-cz2kl\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:50:07.298747 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.298693 2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:50:07.298747 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.298705 2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-cabundle-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:50:07.298747 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.298717 2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50cc4ced-dfcd-41d7-b5b8-0492a98b3406-kserve-provision-location\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:50:07.489036 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.488997 2568 generic.go:356] "Generic (PLEG): container finished" podID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerID="e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3" exitCode=0 Sep 17 18:50:07.489229 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.489046 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerDied","Data":"e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3"} Sep 17 18:50:07.489229 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.489052 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" Sep 17 18:50:07.489229 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.489108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5" event={"ID":"50cc4ced-dfcd-41d7-b5b8-0492a98b3406","Type":"ContainerDied","Data":"47456381b6ba62600d327ef7a73dc0216653fa27768dd6a615e3777e25b10830"} Sep 17 18:50:07.489229 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.489141 2568 scope.go:117] "RemoveContainer" containerID="b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a" Sep 17 18:50:07.499976 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.499778 2568 scope.go:117] "RemoveContainer" containerID="e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3" Sep 17 18:50:07.507539 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.507521 2568 scope.go:117] "RemoveContainer" containerID="03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986" Sep 17 18:50:07.514803 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.514778 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5"] Sep 17 18:50:07.515258 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.515241 2568 scope.go:117] "RemoveContainer" containerID="b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a" Sep 17 18:50:07.515579 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:50:07.515556 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a\": container with ID starting with b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a not found: ID does not exist" containerID="b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a" Sep 17 18:50:07.515627 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.515587 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a"} err="failed to get container status \"b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a\": rpc error: code = NotFound desc = could not find container \"b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a\": container with ID starting with b54e9a4b4611b5fc0b876dcc636a7e5ba526fcc820641b21a326623db1867d4a not found: ID does not exist" Sep 17 18:50:07.515627 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.515605 2568 scope.go:117] "RemoveContainer" containerID="e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3" Sep 17 18:50:07.515871 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:50:07.515854 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3\": container with ID starting with e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3 not found: ID does not exist" containerID="e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3" Sep 17 18:50:07.515912 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.515878 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3"} err="failed to get container status \"e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3\": rpc error: code = NotFound desc = could not find container \"e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3\": container with ID starting with e91ce1a64abf2038f2d9a6c4a3017da5cb5ff89d982c1acaa29c4d5830f358b3 not found: ID does not exist" Sep 17 18:50:07.515912 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.515893 2568 scope.go:117] "RemoveContainer" containerID="03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986" Sep 17 18:50:07.516126 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:50:07.516109 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986\": container with ID starting with 03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986 not found: ID does not exist" containerID="03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986" Sep 17 18:50:07.516175 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.516130 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986"} err="failed to get container status \"03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986\": rpc error: code = NotFound desc = could not find container \"03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986\": container with ID starting with 03a6585fbabdc1d1ed87ad1154ff6c697b67077b02421ab5447ac238fb06b986 not found: ID does not exist" Sep 17 18:50:07.519273 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:07.519248 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a1be8-predictor-5bfcf47-rvpf5"] Sep 17 18:50:08.984139 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:50:08.984107 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" path="/var/lib/kubelet/pods/50cc4ced-dfcd-41d7-b5b8-0492a98b3406/volumes" Sep 17 18:54:14.981051 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:54:14.981012 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:54:14.983942 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:54:14.983912 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:54:14.987585 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:54:14.987563 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:54:14.989957 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:54:14.989935 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:56:40.201658 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.201621 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4ldx/must-gather-s4xxn"] Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202002 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="storage-initializer" Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202016 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="storage-initializer" Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202031 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202037 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202049 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kube-rbac-proxy" Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202055 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kube-rbac-proxy" Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202108 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kube-rbac-proxy" Sep 17 18:56:40.204173 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.202120 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cc4ced-dfcd-41d7-b5b8-0492a98b3406" containerName="kserve-container" Sep 17 18:56:40.205040 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.205020 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.209739 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.209713 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ldx\"/\"kube-root-ca.crt\"" Sep 17 18:56:40.209849 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.209779 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g4ldx\"/\"default-dockercfg-pj8rp\"" Sep 17 18:56:40.209849 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.209713 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ldx\"/\"openshift-service-ca.crt\"" Sep 17 18:56:40.217286 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.217257 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ldx/must-gather-s4xxn"] Sep 17 18:56:40.314194 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.314168 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwhr\" (UniqueName: \"kubernetes.io/projected/27e0bd99-ee60-4998-8731-da4dde6f7cf1-kube-api-access-dbwhr\") pod \"must-gather-s4xxn\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.314357 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.314213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e0bd99-ee60-4998-8731-da4dde6f7cf1-must-gather-output\") pod \"must-gather-s4xxn\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.414732 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.414690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e0bd99-ee60-4998-8731-da4dde6f7cf1-must-gather-output\") pod \"must-gather-s4xxn\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.414888 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.414759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwhr\" (UniqueName: \"kubernetes.io/projected/27e0bd99-ee60-4998-8731-da4dde6f7cf1-kube-api-access-dbwhr\") pod \"must-gather-s4xxn\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.415037 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.415019 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e0bd99-ee60-4998-8731-da4dde6f7cf1-must-gather-output\") pod \"must-gather-s4xxn\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.424767 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.424744 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwhr\" (UniqueName: \"kubernetes.io/projected/27e0bd99-ee60-4998-8731-da4dde6f7cf1-kube-api-access-dbwhr\") pod \"must-gather-s4xxn\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.518773 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.518688 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:56:40.795881 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.795804 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ldx/must-gather-s4xxn"] Sep 17 18:56:40.798999 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:56:40.798957 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27e0bd99_ee60_4998_8731_da4dde6f7cf1.slice/crio-0213214604ed8e880f962e6fa3d2abb80a6e89db8db9845b72217cdfac19b57e WatchSource:0}: Error finding container 0213214604ed8e880f962e6fa3d2abb80a6e89db8db9845b72217cdfac19b57e: Status 404 returned error can't find the container with id 0213214604ed8e880f962e6fa3d2abb80a6e89db8db9845b72217cdfac19b57e Sep 17 18:56:40.800647 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:40.800630 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:56:41.114787 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:41.114706 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" event={"ID":"27e0bd99-ee60-4998-8731-da4dde6f7cf1","Type":"ContainerStarted","Data":"0213214604ed8e880f962e6fa3d2abb80a6e89db8db9845b72217cdfac19b57e"} Sep 17 18:56:47.149094 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:47.149053 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" event={"ID":"27e0bd99-ee60-4998-8731-da4dde6f7cf1","Type":"ContainerStarted","Data":"a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b"} Sep 17 18:56:47.149695 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:47.149106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" event={"ID":"27e0bd99-ee60-4998-8731-da4dde6f7cf1","Type":"ContainerStarted","Data":"5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6"} Sep 17 18:56:47.180332 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:56:47.180277 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" podStartSLOduration=1.146852549 podStartE2EDuration="7.180260619s" totalImagesPullingTime="6.03340807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:47.177817918 +0000 UTC m=+1654.457469293" watchObservedRunningTime="2026-09-17 18:56:47.180260619 +0000 UTC m=+1654.459911993" Sep 17 18:57:10.259611 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:10.259575 2568 generic.go:356] "Generic (PLEG): container finished" podID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerID="5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6" exitCode=0 Sep 17 18:57:10.260020 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:10.259656 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" event={"ID":"27e0bd99-ee60-4998-8731-da4dde6f7cf1","Type":"ContainerDied","Data":"5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6"} Sep 17 18:57:10.260020 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:10.260008 2568 scope.go:117] "RemoveContainer" containerID="5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6" Sep 17 18:57:11.241246 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:11.241216 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g4ldx_must-gather-s4xxn_27e0bd99-ee60-4998-8731-da4dde6f7cf1/gather/0.log" Sep 17 18:57:14.847830 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:14.847796 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6hvl9_8747e254-9790-4236-87a0-9955cb416dde/global-pull-secret-syncer/0.log" Sep 17 18:57:15.004526 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:15.004496 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sbdzm_d63869ab-2129-4c22-bdde-754ea60a7cf6/konnectivity-agent/0.log" Sep 17 18:57:15.055573 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:15.055539 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-202.ec2.internal_1f316321b0ddd050eb379ecf12c03828/haproxy/0.log" Sep 17 18:57:16.814828 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:16.814789 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-g4ldx/must-gather-s4xxn"] Sep 17 18:57:16.815273 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:16.814965 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerName="copy" containerID="cri-o://a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b" gracePeriod=2 Sep 17 18:57:16.821423 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:16.821394 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-g4ldx/must-gather-s4xxn"] Sep 17 18:57:16.823201 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:57:16.823162 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"27e0bd99-ee60-4998-8731-da4dde6f7cf1\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-g4ldx\"/\"must-gather-s4xxn\": pods \"must-gather-s4xxn\" not found" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:57:17.041022 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.040995 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g4ldx_must-gather-s4xxn_27e0bd99-ee60-4998-8731-da4dde6f7cf1/copy/0.log" Sep 17 18:57:17.042586 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.042570 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:57:17.134562 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.134529 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbwhr\" (UniqueName: \"kubernetes.io/projected/27e0bd99-ee60-4998-8731-da4dde6f7cf1-kube-api-access-dbwhr\") pod \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " Sep 17 18:57:17.134728 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.134577 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e0bd99-ee60-4998-8731-da4dde6f7cf1-must-gather-output\") pod \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\" (UID: \"27e0bd99-ee60-4998-8731-da4dde6f7cf1\") " Sep 17 18:57:17.136670 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.136637 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e0bd99-ee60-4998-8731-da4dde6f7cf1-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "27e0bd99-ee60-4998-8731-da4dde6f7cf1" (UID: "27e0bd99-ee60-4998-8731-da4dde6f7cf1"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:57:17.136798 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.136753 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e0bd99-ee60-4998-8731-da4dde6f7cf1-kube-api-access-dbwhr" (OuterVolumeSpecName: "kube-api-access-dbwhr") pod "27e0bd99-ee60-4998-8731-da4dde6f7cf1" (UID: "27e0bd99-ee60-4998-8731-da4dde6f7cf1"). InnerVolumeSpecName "kube-api-access-dbwhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:57:17.235487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.235436 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwhr\" (UniqueName: \"kubernetes.io/projected/27e0bd99-ee60-4998-8731-da4dde6f7cf1-kube-api-access-dbwhr\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:57:17.235487 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.235480 2568 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27e0bd99-ee60-4998-8731-da4dde6f7cf1-must-gather-output\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Sep 17 18:57:17.290020 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.289992 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-g4ldx_must-gather-s4xxn_27e0bd99-ee60-4998-8731-da4dde6f7cf1/copy/0.log" Sep 17 18:57:17.290423 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.290397 2568 generic.go:356] "Generic (PLEG): container finished" podID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerID="a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b" exitCode=143 Sep 17 18:57:17.290538 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.290464 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ldx/must-gather-s4xxn" Sep 17 18:57:17.290538 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.290495 2568 scope.go:117] "RemoveContainer" containerID="a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b" Sep 17 18:57:17.298327 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.298300 2568 scope.go:117] "RemoveContainer" containerID="5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6" Sep 17 18:57:17.309681 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.309659 2568 scope.go:117] "RemoveContainer" containerID="a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b" Sep 17 18:57:17.309947 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:57:17.309931 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b\": container with ID starting with a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b not found: ID does not exist" containerID="a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b" Sep 17 18:57:17.310007 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.309954 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b"} err="failed to get container status \"a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b\": rpc error: code = NotFound desc = could not find container \"a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b\": container with ID starting with a60c63784879be5895a00bcfc30597d553e6b6b210d7347f41503099a847800b not found: ID does not exist" Sep 17 18:57:17.310007 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.309971 2568 scope.go:117] "RemoveContainer" containerID="5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6" Sep 17 18:57:17.310203 ip-10-0-134-202 kubenswrapper[2568]: E0917 18:57:17.310188 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6\": container with ID starting with 5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6 not found: ID does not exist" containerID="5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6" Sep 17 18:57:17.310245 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:17.310208 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6"} err="failed to get container status \"5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6\": rpc error: code = NotFound desc = could not find container \"5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6\": container with ID starting with 5cf46e78a4ea3a1ae4c517a68fa65f5ce7000be5fcb9b9af521112d5e5fdd1e6 not found: ID does not exist" Sep 17 18:57:18.984120 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:18.984088 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" path="/var/lib/kubelet/pods/27e0bd99-ee60-4998-8731-da4dde6f7cf1/volumes" Sep 17 18:57:19.140456 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.140408 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-h8rn8_d4c66ecd-c5cf-4ea9-9674-edfbf632764d/cluster-monitoring-operator/0.log" Sep 17 18:57:19.170156 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.170120 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-t2lzq_3f93f2df-5aba-4ee6-9125-bb7370fdc5c4/kube-state-metrics/0.log" Sep 17 18:57:19.196780 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.196750 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-t2lzq_3f93f2df-5aba-4ee6-9125-bb7370fdc5c4/kube-rbac-proxy-main/0.log" Sep 17 18:57:19.230648 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.230556 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-t2lzq_3f93f2df-5aba-4ee6-9125-bb7370fdc5c4/kube-rbac-proxy-self/0.log" Sep 17 18:57:19.296985 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.296959 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-p4tpd_a84c96cf-3f47-49d7-bd1e-0651f3169664/monitoring-plugin/0.log" Sep 17 18:57:19.526778 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.526684 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqjr6_fe52b640-325a-4086-93ad-8d0829be056a/node-exporter/0.log" Sep 17 18:57:19.556896 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.556868 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqjr6_fe52b640-325a-4086-93ad-8d0829be056a/kube-rbac-proxy/0.log" Sep 17 18:57:19.580761 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.580736 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vqjr6_fe52b640-325a-4086-93ad-8d0829be056a/init-textfile/0.log" Sep 17 18:57:19.714646 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.714605 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93c872b5-842d-4bd9-8f85-4a6014d169f8/prometheus/0.log" Sep 17 18:57:19.753360 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.753333 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93c872b5-842d-4bd9-8f85-4a6014d169f8/config-reloader/0.log" Sep 17 18:57:19.779307 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.779226 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93c872b5-842d-4bd9-8f85-4a6014d169f8/thanos-sidecar/0.log" Sep 17 18:57:19.803484 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.803457 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93c872b5-842d-4bd9-8f85-4a6014d169f8/kube-rbac-proxy-web/0.log" Sep 17 18:57:19.836693 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.836658 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93c872b5-842d-4bd9-8f85-4a6014d169f8/kube-rbac-proxy/0.log" Sep 17 18:57:19.864020 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.863996 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93c872b5-842d-4bd9-8f85-4a6014d169f8/kube-rbac-proxy-thanos/0.log" Sep 17 18:57:19.893984 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:19.893957 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_93c872b5-842d-4bd9-8f85-4a6014d169f8/init-config-reloader/0.log" Sep 17 18:57:20.093570 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:20.093531 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6f75df8-fjmnm_8683d8bc-3ce6-4dfc-be72-d86c2890ba21/thanos-query/0.log" Sep 17 18:57:20.116317 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:20.116276 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6f75df8-fjmnm_8683d8bc-3ce6-4dfc-be72-d86c2890ba21/kube-rbac-proxy-web/0.log" Sep 17 18:57:20.141481 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:20.141437 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6f75df8-fjmnm_8683d8bc-3ce6-4dfc-be72-d86c2890ba21/kube-rbac-proxy/0.log" Sep 17 18:57:20.165085 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:20.165055 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6f75df8-fjmnm_8683d8bc-3ce6-4dfc-be72-d86c2890ba21/prom-label-proxy/0.log" Sep 17 18:57:20.188005 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:20.187976 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6f75df8-fjmnm_8683d8bc-3ce6-4dfc-be72-d86c2890ba21/kube-rbac-proxy-rules/0.log" Sep 17 18:57:20.214315 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:20.214291 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c6f75df8-fjmnm_8683d8bc-3ce6-4dfc-be72-d86c2890ba21/kube-rbac-proxy-metrics/0.log" Sep 17 18:57:21.562831 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:21.562743 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-hl4kd_b4f7c3c5-0275-4c94-a7d7-0c7bc64806b6/networking-console-plugin/0.log" Sep 17 18:57:22.126620 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:22.126591 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/1.log" Sep 17 18:57:22.136210 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:22.136183 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-ssmlh_0f3e7043-0001-4c19-aefd-19d656d570fd/console-operator/2.log" Sep 17 18:57:23.244559 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.244527 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg"] Sep 17 18:57:23.244943 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.244865 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerName="copy" Sep 17 18:57:23.244943 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.244876 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerName="copy" Sep 17 18:57:23.244943 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.244893 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerName="gather" Sep 17 18:57:23.244943 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.244898 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerName="gather" Sep 17 18:57:23.245080 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.244958 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerName="copy" Sep 17 18:57:23.245080 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.244968 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e0bd99-ee60-4998-8731-da4dde6f7cf1" containerName="gather" Sep 17 18:57:23.248188 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.248170 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.251084 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.251062 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6x4n\"/\"kube-root-ca.crt\"" Sep 17 18:57:23.251891 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.251875 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b6x4n\"/\"default-dockercfg-tjq4r\"" Sep 17 18:57:23.251950 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.251911 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6x4n\"/\"openshift-service-ca.crt\"" Sep 17 18:57:23.257904 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.257881 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg"] Sep 17 18:57:23.265761 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.265735 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-b8vdf_68273c1d-c8b7-4149-a872-0ca8a179333d/volume-data-source-validator/0.log" Sep 17 18:57:23.283062 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.283031 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-podres\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.283062 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.283063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbqn\" (UniqueName: \"kubernetes.io/projected/cae79547-b279-48e3-8e92-3617947b1be1-kube-api-access-hjbqn\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.283251 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.283105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-lib-modules\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.283251 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.283160 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-sys\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.283251 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.283206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-proc\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384138 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384102 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-proc\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384291 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-podres\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384291 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbqn\" (UniqueName: \"kubernetes.io/projected/cae79547-b279-48e3-8e92-3617947b1be1-kube-api-access-hjbqn\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384291 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384231 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-proc\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384291 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384242 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-lib-modules\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384291 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-sys\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384530 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-podres\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384530 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-sys\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.384530 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.384416 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cae79547-b279-48e3-8e92-3617947b1be1-lib-modules\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.393355 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.393329 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbqn\" (UniqueName: \"kubernetes.io/projected/cae79547-b279-48e3-8e92-3617947b1be1-kube-api-access-hjbqn\") pod \"perf-node-gather-daemonset-hhmqg\" (UID: \"cae79547-b279-48e3-8e92-3617947b1be1\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.558123 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.558035 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:23.827584 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:23.827501 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg"] Sep 17 18:57:23.831411 ip-10-0-134-202 kubenswrapper[2568]: W0917 18:57:23.831384 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcae79547_b279_48e3_8e92_3617947b1be1.slice/crio-c3b4ee9c111fcc6153b474619029f5e7cc417018432c2f5bc6dd2b4d16431558 WatchSource:0}: Error finding container c3b4ee9c111fcc6153b474619029f5e7cc417018432c2f5bc6dd2b4d16431558: Status 404 returned error can't find the container with id c3b4ee9c111fcc6153b474619029f5e7cc417018432c2f5bc6dd2b4d16431558 Sep 17 18:57:24.230480 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.230437 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbkr4_1cc0cf8d-03e2-489e-9a01-cb24fa4ee382/dns/0.log" Sep 17 18:57:24.256142 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.256117 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbkr4_1cc0cf8d-03e2-489e-9a01-cb24fa4ee382/kube-rbac-proxy/0.log" Sep 17 18:57:24.325564 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.325529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" event={"ID":"cae79547-b279-48e3-8e92-3617947b1be1","Type":"ContainerStarted","Data":"7c2ecf5f3e0485f420ac2ae5c9a62e0e2f4439a62643cb5176a5d5e6ce270ce3"} Sep 17 18:57:24.325743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.325584 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" event={"ID":"cae79547-b279-48e3-8e92-3617947b1be1","Type":"ContainerStarted","Data":"c3b4ee9c111fcc6153b474619029f5e7cc417018432c2f5bc6dd2b4d16431558"} Sep 17 18:57:24.325743 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.325608 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:24.342574 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.342518 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" podStartSLOduration=1.342497185 podStartE2EDuration="1.342497185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:24.342404797 +0000 UTC m=+1691.622056173" watchObservedRunningTime="2026-09-17 18:57:24.342497185 +0000 UTC m=+1691.622148559" Sep 17 18:57:24.364595 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.364569 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-75d5b_c672c626-1477-4611-a266-16e533c89d87/dns-node-resolver/0.log" Sep 17 18:57:24.904951 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:24.904908 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vhft5_092bdb7d-656a-4900-8af6-f0a03eab5aa9/node-ca/0.log" Sep 17 18:57:25.733872 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:25.733845 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-695f977464-jzv52_d6f4e7bd-7144-495f-b55c-08c3a876a3ad/router/0.log" Sep 17 18:57:26.105742 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:26.105656 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rnxzj_5b748973-196a-4cf2-a06f-67b2588fc4ef/serve-healthcheck-canary/0.log" Sep 17 18:57:26.456857 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:26.456772 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-x9qpw_7869cb96-52c3-46c3-b799-1412fbce679b/insights-operator/0.log" Sep 17 18:57:26.458129 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:26.458098 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-x9qpw_7869cb96-52c3-46c3-b799-1412fbce679b/insights-operator/1.log" Sep 17 18:57:26.545789 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:26.545747 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g46wv_7f030f12-0b17-4c45-bfb4-c428a1788ed9/kube-rbac-proxy/0.log" Sep 17 18:57:26.567170 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:26.567140 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g46wv_7f030f12-0b17-4c45-bfb4-c428a1788ed9/exporter/0.log" Sep 17 18:57:26.588861 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:26.588821 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g46wv_7f030f12-0b17-4c45-bfb4-c428a1788ed9/extractor/0.log" Sep 17 18:57:29.178015 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:29.177980 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8gkz5_092811a0-fae6-4ab6-a753-1f1dc6307337/s3-init/0.log" Sep 17 18:57:29.201666 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:29.201631 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8gkz5_092811a0-fae6-4ab6-a753-1f1dc6307337/download-hf-model/0.log" Sep 17 18:57:30.339418 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:30.339393 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-hhmqg" Sep 17 18:57:33.920618 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:33.920586 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-kt2hg_fa1cff07-b71c-4c36-b79d-5376eb90acdd/kube-storage-version-migrator-operator/1.log" Sep 17 18:57:33.922325 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:33.922294 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-kt2hg_fa1cff07-b71c-4c36-b79d-5376eb90acdd/kube-storage-version-migrator-operator/0.log" Sep 17 18:57:35.409984 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.409955 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbgsj_2b74e7b8-7738-4fc8-9b08-27179c470990/kube-multus-additional-cni-plugins/0.log" Sep 17 18:57:35.444530 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.444495 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbgsj_2b74e7b8-7738-4fc8-9b08-27179c470990/egress-router-binary-copy/0.log" Sep 17 18:57:35.476241 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.476215 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbgsj_2b74e7b8-7738-4fc8-9b08-27179c470990/cni-plugins/0.log" Sep 17 18:57:35.499851 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.499824 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbgsj_2b74e7b8-7738-4fc8-9b08-27179c470990/bond-cni-plugin/0.log" Sep 17 18:57:35.535065 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.535035 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbgsj_2b74e7b8-7738-4fc8-9b08-27179c470990/routeoverride-cni/0.log" Sep 17 18:57:35.564551 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.564519 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbgsj_2b74e7b8-7738-4fc8-9b08-27179c470990/whereabouts-cni-bincopy/0.log" Sep 17 18:57:35.588013 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.587979 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbgsj_2b74e7b8-7738-4fc8-9b08-27179c470990/whereabouts-cni/0.log" Sep 17 18:57:35.627359 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.627327 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r82qp_e3d8596e-c089-4b0b-89ab-68002f3c78d8/kube-multus/0.log" Sep 17 18:57:35.796705 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.796620 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zq6bz_370e5c7b-6a22-41a2-bbe8-67e7f11bc373/network-metrics-daemon/0.log" Sep 17 18:57:35.821419 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:35.821364 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zq6bz_370e5c7b-6a22-41a2-bbe8-67e7f11bc373/kube-rbac-proxy/0.log" Sep 17 18:57:36.615827 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.615796 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-controller/0.log" Sep 17 18:57:36.658502 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.658471 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/0.log" Sep 17 18:57:36.658692 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.658470 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovn-acl-logging/1.log" Sep 17 18:57:36.697948 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.697915 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/kube-rbac-proxy-node/0.log" Sep 17 18:57:36.765055 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.765022 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/kube-rbac-proxy-ovn-metrics/0.log" Sep 17 18:57:36.803590 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.803558 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/northd/0.log" Sep 17 18:57:36.831930 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.831889 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/nbdb/0.log" Sep 17 18:57:36.862113 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.862083 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/sbdb/0.log" Sep 17 18:57:36.921572 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:36.921537 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-895jp_52d928d4-4882-46ee-91c9-47e723e32b3a/ovnkube-controller/0.log" Sep 17 18:57:38.410287 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:38.410253 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-6fv49_e1782d59-4fd9-4076-b0f4-54067c4cfd4c/check-endpoints/0.log" Sep 17 18:57:38.483701 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:38.483670 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wh8zt_00162636-5bc2-435a-8fbe-39f28f63641d/network-check-target-container/0.log" Sep 17 18:57:39.442661 ip-10-0-134-202 kubenswrapper[2568]: I0917 18:57:39.442574 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mrpkj_01a5d8fc-8768-4c7d-971a-485e2cd09de7/iptables-alerter/0.log"