May 25 13:06:28.205697 ip-10-0-135-64 systemd[1]: Starting Kubernetes Kubelet... May 25 13:06:28.650608 ip-10-0-135-64 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 25 13:06:28.650608 ip-10-0-135-64 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 25 13:06:28.650608 ip-10-0-135-64 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 25 13:06:28.650608 ip-10-0-135-64 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 25 13:06:28.650608 ip-10-0-135-64 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 25 13:06:28.652198 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.652126 2554 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 25 13:06:28.658384 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658370 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 13:06:28.658384 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658385 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658389 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658392 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658396 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658399 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658402 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658404 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658407 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658410 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658413 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658419 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658433 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658436 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658438 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658441 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658444 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658447 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658449 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658454 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 13:06:28.658455 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658457 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658460 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658463 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658466 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658469 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658472 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658474 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658477 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658480 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658482 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658485 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658487 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658490 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658492 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658495 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658497 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658500 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658503 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658505 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658508 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 13:06:28.658904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658510 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658513 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658515 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658519 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658521 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658524 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658526 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658529 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658531 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658534 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658536 2554 feature_gate.go:328] unrecognized feature gate: Example May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658539 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658541 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658544 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658547 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658550 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658553 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658555 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658558 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658560 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 13:06:28.659378 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658562 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658565 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658567 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658570 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658573 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658577 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658580 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658583 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658586 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658588 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658591 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658593 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658596 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658598 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658601 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658604 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658606 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658609 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658612 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 25 13:06:28.659904 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658615 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658617 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658620 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658623 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658625 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658628 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.658630 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659010 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659015 2554 feature_gate.go:328] unrecognized feature gate: Example May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659018 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659021 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659023 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659026 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659028 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659031 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659034 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659036 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659039 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659042 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659045 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 13:06:28.660372 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659048 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659050 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659053 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659055 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659058 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659060 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659063 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659065 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659068 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659071 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659074 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659076 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659079 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659081 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659084 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659087 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659091 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659095 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659097 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 13:06:28.660885 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659101 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659104 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659106 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659109 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659111 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659113 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659116 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659119 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659122 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659124 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659127 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659129 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659131 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659134 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659137 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659139 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659142 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659145 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659147 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659150 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 13:06:28.661354 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659153 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659155 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659158 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659160 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659163 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659166 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659168 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659170 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659173 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659176 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659179 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659181 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659184 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659187 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659189 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659193 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659197 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659200 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659203 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 13:06:28.661887 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659206 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659208 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659211 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659213 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659216 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659218 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659221 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659223 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659225 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659228 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659230 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659232 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659235 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659237 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659240 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659309 2554 flags.go:64] FLAG: --address="0.0.0.0" May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659317 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659323 2554 flags.go:64] FLAG: --anonymous-auth="true" May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659328 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659332 2554 flags.go:64] FLAG: --authentication-token-webhook="false" May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659335 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 25 13:06:28.662356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659340 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659344 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659347 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659351 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659354 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659358 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659361 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659364 2554 flags.go:64] FLAG: --cgroup-root="" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659367 2554 flags.go:64] FLAG: --cgroups-per-qos="true" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659370 2554 flags.go:64] FLAG: --client-ca-file="" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659373 2554 flags.go:64] FLAG: --cloud-config="" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659376 2554 flags.go:64] FLAG: --cloud-provider="external" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659379 2554 flags.go:64] FLAG: --cluster-dns="[]" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659383 2554 flags.go:64] FLAG: --cluster-domain="" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659386 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659389 2554 flags.go:64] FLAG: --config-dir="" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659392 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659395 2554 flags.go:64] FLAG: --container-log-max-files="5" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659399 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659402 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659405 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659408 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659411 2554 flags.go:64] FLAG: --contention-profiling="false" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659414 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" May 25 13:06:28.662899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659417 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659433 2554 flags.go:64] FLAG: --cpu-manager-policy="none" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659436 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659440 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659443 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659446 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659450 2554 flags.go:64] FLAG: --enable-load-reader="false" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659453 2554 flags.go:64] FLAG: --enable-server="true" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659456 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659460 2554 flags.go:64] FLAG: --event-burst="100" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659463 2554 flags.go:64] FLAG: --event-qps="50" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659466 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659469 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659472 2554 flags.go:64] FLAG: --eviction-hard="" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659476 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659478 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659481 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659485 2554 flags.go:64] FLAG: --eviction-soft="" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659487 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659490 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659493 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659496 2554 flags.go:64] FLAG: --experimental-mounter-path="" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659499 2554 flags.go:64] FLAG: --fail-cgroupv1="false" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659502 2554 flags.go:64] FLAG: --fail-swap-on="true" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659505 2554 flags.go:64] FLAG: --feature-gates="" May 25 13:06:28.663506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659508 2554 flags.go:64] FLAG: --file-check-frequency="20s" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659511 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659514 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659518 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659521 2554 flags.go:64] FLAG: --healthz-port="10248" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659524 2554 flags.go:64] FLAG: --help="false" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659526 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-135-64.ec2.internal" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659529 2554 flags.go:64] FLAG: --housekeeping-interval="10s" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659532 2554 flags.go:64] FLAG: --http-check-frequency="20s" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659535 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659538 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659542 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659545 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659547 2554 flags.go:64] FLAG: --image-service-endpoint="" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659551 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659554 2554 flags.go:64] FLAG: --kube-api-burst="100" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659557 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659560 2554 flags.go:64] FLAG: --kube-api-qps="50" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659563 2554 flags.go:64] FLAG: --kube-reserved="" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659566 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659569 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659572 2554 flags.go:64] FLAG: --kubelet-cgroups="" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659575 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659578 2554 flags.go:64] FLAG: --lock-file="" May 25 13:06:28.664119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659581 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659584 2554 flags.go:64] FLAG: --log-flush-frequency="5s" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659586 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659591 2554 flags.go:64] FLAG: --log-json-split-stream="false" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659594 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659597 2554 flags.go:64] FLAG: --log-text-split-stream="false" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659600 2554 flags.go:64] FLAG: --logging-format="text" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659603 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659606 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659609 2554 flags.go:64] FLAG: --manifest-url="" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659612 2554 flags.go:64] FLAG: --manifest-url-header="" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659616 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659619 2554 flags.go:64] FLAG: --max-open-files="1000000" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659623 2554 flags.go:64] FLAG: --max-pods="110" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659625 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659628 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659631 2554 flags.go:64] FLAG: --memory-manager-policy="None" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659634 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659637 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659640 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659643 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659651 2554 flags.go:64] FLAG: --node-status-max-images="50" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659654 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659657 2554 flags.go:64] FLAG: --oom-score-adj="-999" May 25 13:06:28.664734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659660 2554 flags.go:64] FLAG: --pod-cidr="" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659663 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659669 2554 flags.go:64] FLAG: --pod-manifest-path="" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659672 2554 flags.go:64] FLAG: --pod-max-pids="-1" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659675 2554 flags.go:64] FLAG: --pods-per-core="0" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659678 2554 flags.go:64] FLAG: --port="10250" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659682 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659684 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0058509e2906bff98" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659688 2554 flags.go:64] FLAG: --qos-reserved="" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659691 2554 flags.go:64] FLAG: --read-only-port="10255" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659693 2554 flags.go:64] FLAG: --register-node="true" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659697 2554 flags.go:64] FLAG: --register-schedulable="true" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659699 2554 flags.go:64] FLAG: --register-with-taints="" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659703 2554 flags.go:64] FLAG: --registry-burst="10" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659706 2554 flags.go:64] FLAG: --registry-qps="5" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659708 2554 flags.go:64] FLAG: --reserved-cpus="" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659711 2554 flags.go:64] FLAG: --reserved-memory="" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659715 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659718 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659721 2554 flags.go:64] FLAG: --rotate-certificates="false" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659724 2554 flags.go:64] FLAG: --rotate-server-certificates="false" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659727 2554 flags.go:64] FLAG: --runonce="false" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659730 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659733 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659735 2554 flags.go:64] FLAG: --seccomp-default="false" May 25 13:06:28.665306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659738 2554 flags.go:64] FLAG: --serialize-image-pulls="true" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659741 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659746 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659749 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659752 2554 flags.go:64] FLAG: --storage-driver-password="root" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659755 2554 flags.go:64] FLAG: --storage-driver-secure="false" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659758 2554 flags.go:64] FLAG: --storage-driver-table="stats" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659765 2554 flags.go:64] FLAG: --storage-driver-user="root" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659768 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659771 2554 flags.go:64] FLAG: --sync-frequency="1m0s" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659774 2554 flags.go:64] FLAG: --system-cgroups="" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659776 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659782 2554 flags.go:64] FLAG: --system-reserved-cgroup="" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659784 2554 flags.go:64] FLAG: --tls-cert-file="" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659787 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659792 2554 flags.go:64] FLAG: --tls-min-version="" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659794 2554 flags.go:64] FLAG: --tls-private-key-file="" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659797 2554 flags.go:64] FLAG: --topology-manager-policy="none" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659800 2554 flags.go:64] FLAG: --topology-manager-policy-options="" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659803 2554 flags.go:64] FLAG: --topology-manager-scope="container" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659806 2554 flags.go:64] FLAG: --v="2" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659810 2554 flags.go:64] FLAG: --version="false" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659813 2554 flags.go:64] FLAG: --vmodule="" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659818 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.659821 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 25 13:06:28.665941 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659913 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659917 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659919 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659922 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659925 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659927 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659931 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659935 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659938 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659942 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659945 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659948 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659950 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659953 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659958 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659961 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659964 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659966 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659969 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659971 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 13:06:28.666596 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659974 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659976 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659979 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659981 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659984 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659987 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659989 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659991 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659994 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659996 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.659999 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660001 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660004 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660006 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660009 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660012 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660014 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660017 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660019 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660022 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 25 13:06:28.667379 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660024 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660028 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660031 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660034 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660037 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660040 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660043 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660046 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660049 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660052 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660055 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660057 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660060 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660062 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660064 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660067 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660069 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660072 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660074 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660077 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 13:06:28.667974 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660079 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660082 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660084 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660087 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660089 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660091 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660094 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660096 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660099 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660102 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660104 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660106 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660109 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660112 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660115 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660117 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660120 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660123 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660126 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 13:06:28.668544 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660130 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660134 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660137 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660140 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660143 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660145 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.660148 2554 feature_gate.go:328] unrecognized feature gate: Example May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.660153 2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.668912 2554 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 25 13:06:28.669018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.669010 2554 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669058 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669063 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669066 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669069 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669072 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669075 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669078 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669081 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669083 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669086 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669089 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669092 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669094 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669097 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669099 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669104 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669108 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669112 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 13:06:28.669246 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669115 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669118 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669121 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669124 2554 feature_gate.go:328] unrecognized feature gate: Example May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669127 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669130 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669133 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669136 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669139 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669141 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669145 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669147 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669150 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669153 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669155 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669158 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669161 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669164 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669166 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 13:06:28.669725 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669171 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669174 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669177 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669180 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669182 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669185 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669187 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669190 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669193 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669195 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669198 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669201 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669203 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669206 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669209 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669211 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669214 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669216 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669219 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669222 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 13:06:28.670191 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669225 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669228 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669231 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669233 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669236 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669238 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669241 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669244 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669247 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669249 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669252 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669255 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669258 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669261 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669263 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669266 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669268 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669271 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669274 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669277 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 13:06:28.670720 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669279 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669282 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669285 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669287 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669290 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669292 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669295 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669297 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669300 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.669305 2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669398 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669404 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669407 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669410 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669413 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669416 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 25 13:06:28.671204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669419 2554 feature_gate.go:328] unrecognized feature gate: NewOLM May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669441 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669445 2554 feature_gate.go:328] unrecognized feature gate: Example May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669448 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669451 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669454 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669456 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669459 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669461 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669464 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669467 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669469 2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669472 2554 feature_gate.go:328] unrecognized feature gate: Example2 May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669475 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669477 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669480 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669483 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669486 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669490 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 25 13:06:28.671615 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669492 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669495 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669498 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669500 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669503 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669506 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669509 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669511 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669514 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669516 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669520 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669522 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669525 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669528 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669531 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669534 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669536 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669539 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669542 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669544 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages May 25 13:06:28.672076 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669547 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669549 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669552 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669554 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669557 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669560 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669562 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669566 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669569 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669572 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669576 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669578 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669581 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669584 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669587 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669589 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669592 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669594 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669597 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 25 13:06:28.672591 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669599 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669602 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669604 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669607 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669610 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669612 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669615 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669617 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669620 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669623 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669625 2554 feature_gate.go:328] unrecognized feature gate: DualReplica May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669628 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669630 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669633 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669635 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669638 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669640 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669643 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669645 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669648 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 25 13:06:28.673045 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669650 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 25 13:06:28.673590 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:28.669653 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 25 13:06:28.673590 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.669657 2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 25 13:06:28.673590 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.670370 2554 server.go:962] "Client rotation is on, will bootstrap in background" May 25 13:06:28.673590 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.673243 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 25 13:06:28.674235 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.674223 2554 server.go:1019] "Starting client certificate rotation" May 25 13:06:28.674334 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.674318 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 25 13:06:28.674370 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.674356 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 25 13:06:28.697770 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.697753 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 25 13:06:28.701261 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.701239 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 25 13:06:28.715179 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.715163 2554 log.go:25] "Validated CRI v1 runtime API" May 25 13:06:28.724344 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.724316 2554 log.go:25] "Validated CRI v1 image API" May 25 13:06:28.725568 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.725552 2554 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 25 13:06:28.730006 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.729987 2554 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 25 13:06:28.731356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.731335 2554 fs.go:135] Filesystem UUIDs: map[2412a7af-d9b8-4615-a21e-39e1f7b5deb2:/dev/nvme0n1p4 715053ac-53ba-4c9e-882a-fa9bb69bc044:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] May 25 13:06:28.731457 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.731356 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 25 13:06:28.735617 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.735508 2554 manager.go:217] Machine: {Timestamp:2026-05-25 13:06:28.735019851 +0000 UTC m=+0.413317444 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099704 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b029b152c5b408928c56756c3b88 SystemUUID:ec21b029-b152-c5b4-0892-8c56756c3b88 BootID:dc5a7557-28fd-4bfc-b928-f10d55aa092c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a6:09:64:38:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a6:09:64:38:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:c0:48:92:b9:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 25 13:06:28.736291 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.736278 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 25 13:06:28.736446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.736413 2554 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 25 13:06:28.740065 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.740043 2554 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 25 13:06:28.740192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.740068 2554 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 25 13:06:28.740234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.740203 2554 topology_manager.go:138] "Creating topology manager with none policy" May 25 13:06:28.740234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.740211 2554 container_manager_linux.go:306] "Creating device plugin manager" May 25 13:06:28.740234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.740218 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 25 13:06:28.741011 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.741001 2554 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 25 13:06:28.742235 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.742225 2554 state_mem.go:36] "Initialized new in-memory state store" May 25 13:06:28.742334 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.742326 2554 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 25 13:06:28.744597 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.744587 2554 kubelet.go:491] "Attempting to sync node with API server" May 25 13:06:28.744631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.744601 2554 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 25 13:06:28.744631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.744612 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 25 13:06:28.744631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.744621 2554 kubelet.go:397] "Adding apiserver pod source" May 25 13:06:28.744631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.744629 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 25 13:06:28.745648 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.745636 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 25 13:06:28.745686 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.745655 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 25 13:06:28.749046 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.749031 2554 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 25 13:06:28.750631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.750618 2554 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 25 13:06:28.752459 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752444 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752464 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752471 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752476 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752485 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752492 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752498 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752503 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 25 13:06:28.752510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752509 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 25 13:06:28.752712 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752515 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 25 13:06:28.752712 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752529 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 25 13:06:28.752712 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.752534 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 25 13:06:28.753350 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.753341 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 25 13:06:28.753350 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.753350 2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 25 13:06:28.756647 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.756634 2554 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 25 13:06:28.756708 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.756674 2554 server.go:1295] "Started kubelet" May 25 13:06:28.756767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.756734 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 25 13:06:28.756843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.756773 2554 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 25 13:06:28.756911 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.756867 2554 server_v1.go:47] "podresources" method="list" useActivePods=true May 25 13:06:28.757462 ip-10-0-135-64 systemd[1]: Started Kubernetes Kubelet. May 25 13:06:28.758229 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.758211 2554 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 25 13:06:28.758627 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.758611 2554 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 25 13:06:28.758773 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.758607 2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 25 13:06:28.758773 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.758605 2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 25 13:06:28.760102 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.760078 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-swwdf" May 25 13:06:28.760542 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.760518 2554 server.go:317] "Adding debug handlers to kubelet server" May 25 13:06:28.764252 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.764234 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 25 13:06:28.764463 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.763575 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-64.ec2.internal.18b2d0cea77dcab7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-64.ec2.internal,UID:ip-10-0-135-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-64.ec2.internal,},FirstTimestamp:2026-05-25 13:06:28.756646583 +0000 UTC m=+0.434944176,LastTimestamp:2026-05-25 13:06:28.756646583 +0000 UTC m=+0.434944176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-64.ec2.internal,}" May 25 13:06:28.765497 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.765479 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 25 13:06:28.766516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.765978 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-swwdf" May 25 13:06:28.766516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.766042 2554 volume_manager.go:295] "The desired_state_of_world populator starts" May 25 13:06:28.766516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.766054 2554 volume_manager.go:297] "Starting Kubelet Volume Manager" May 25 13:06:28.766516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.766463 2554 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 25 13:06:28.766703 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.766564 2554 reconstruct.go:97] "Volume reconstruction finished" May 25 13:06:28.766703 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.766572 2554 reconciler.go:26] "Reconciler: start to sync state" May 25 13:06:28.767303 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.767277 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:28.767709 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.767691 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 25 13:06:28.767709 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.767711 2554 factory.go:55] Registering systemd factory May 25 13:06:28.767846 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.767721 2554 factory.go:223] Registration of the systemd container factory successfully May 25 13:06:28.767979 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.767963 2554 factory.go:153] Registering CRI-O factory May 25 13:06:28.768037 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.767984 2554 factory.go:223] Registration of the crio container factory successfully May 25 13:06:28.768037 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.768003 2554 factory.go:103] Registering Raw factory May 25 13:06:28.768037 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.768024 2554 manager.go:1196] Started watching for new ooms in manager May 25 13:06:28.768480 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.768465 2554 manager.go:319] Starting recovery of all containers May 25 13:06:28.771271 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.771245 2554 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 25 13:06:28.773368 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.773352 2554 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 25 13:06:28.775932 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.775911 2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-64.ec2.internal\" not found" node="ip-10-0-135-64.ec2.internal" May 25 13:06:28.779686 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.779667 2554 manager.go:324] Recovery completed May 25 13:06:28.783597 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.783586 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 13:06:28.785313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.785297 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientMemory" May 25 13:06:28.785376 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.785327 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasNoDiskPressure" May 25 13:06:28.785376 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.785338 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientPID" May 25 13:06:28.785815 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.785800 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" May 25 13:06:28.785885 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.785814 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 25 13:06:28.785885 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.785830 2554 state_mem.go:36] "Initialized new in-memory state store" May 25 13:06:28.787802 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.787790 2554 policy_none.go:49] "None policy: Start" May 25 13:06:28.787859 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.787805 2554 memory_manager.go:186] "Starting memorymanager" policy="None" May 25 13:06:28.787859 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.787815 2554 state_mem.go:35] "Initializing new in-memory state store" May 25 13:06:28.831985 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.831967 2554 manager.go:341] "Starting Device Plugin manager" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.832021 2554 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.832030 2554 server.go:85] "Starting device plugin registration server" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.832198 2554 eviction_manager.go:189] "Eviction manager: starting control loop" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.832206 2554 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.832360 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.832461 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.832474 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.832921 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 25 13:06:28.835768 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.832951 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:28.894805 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.894780 2554 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 25 13:06:28.895930 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.895909 2554 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 25 13:06:28.896000 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.895936 2554 status_manager.go:230] "Starting to sync pod status with apiserver" May 25 13:06:28.896000 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.895952 2554 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 25 13:06:28.896000 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.895958 2554 kubelet.go:2451] "Starting kubelet main sync loop" May 25 13:06:28.896000 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.895994 2554 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 25 13:06:28.899090 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.899071 2554 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 25 13:06:28.932810 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.932773 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 13:06:28.934584 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.934571 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientMemory" May 25 13:06:28.934655 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.934597 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasNoDiskPressure" May 25 13:06:28.934655 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.934607 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientPID" May 25 13:06:28.934655 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.934628 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-64.ec2.internal" May 25 13:06:28.942292 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.942277 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-64.ec2.internal" May 25 13:06:28.942339 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.942300 2554 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-64.ec2.internal\": node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:28.962894 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:28.962877 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:28.997089 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.997071 2554 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal"] May 25 13:06:28.997153 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.997123 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 13:06:28.997885 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.997871 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientMemory" May 25 13:06:28.997952 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.997907 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasNoDiskPressure" May 25 13:06:28.997952 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.997920 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientPID" May 25 13:06:28.999028 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999016 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 13:06:28.999220 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999207 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:28.999279 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999239 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 13:06:28.999713 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999700 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientMemory" May 25 13:06:28.999785 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999702 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientMemory" May 25 13:06:28.999785 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999742 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasNoDiskPressure" May 25 13:06:28.999785 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999752 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientPID" May 25 13:06:28.999871 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999722 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasNoDiskPressure" May 25 13:06:28.999871 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:28.999813 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientPID" May 25 13:06:29.000774 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.000759 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" May 25 13:06:29.000843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.000781 2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 25 13:06:29.001924 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.001909 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientMemory" May 25 13:06:29.002018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.001936 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasNoDiskPressure" May 25 13:06:29.002018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.001970 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeHasSufficientPID" May 25 13:06:29.019992 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.019976 2554 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-64.ec2.internal\" not found" node="ip-10-0-135-64.ec2.internal" May 25 13:06:29.025546 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.025530 2554 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-64.ec2.internal\" not found" node="ip-10-0-135-64.ec2.internal" May 25 13:06:29.062939 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.062912 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.068163 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.068143 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3adf5aa3c0fea559adbe132dae0b47e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal\" (UID: \"e3adf5aa3c0fea559adbe132dae0b47e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:29.068213 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.068191 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3adf5aa3c0fea559adbe132dae0b47e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal\" (UID: \"e3adf5aa3c0fea559adbe132dae0b47e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:29.068263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.068220 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48fb10cd0fb028145d43ec292fc620af-config\") pod \"kube-apiserver-proxy-ip-10-0-135-64.ec2.internal\" (UID: \"48fb10cd0fb028145d43ec292fc620af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" May 25 13:06:29.163449 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.163410 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.168670 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.168655 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3adf5aa3c0fea559adbe132dae0b47e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal\" (UID: \"e3adf5aa3c0fea559adbe132dae0b47e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:29.168730 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.168718 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3adf5aa3c0fea559adbe132dae0b47e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal\" (UID: \"e3adf5aa3c0fea559adbe132dae0b47e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:29.168767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.168734 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48fb10cd0fb028145d43ec292fc620af-config\") pod \"kube-apiserver-proxy-ip-10-0-135-64.ec2.internal\" (UID: \"48fb10cd0fb028145d43ec292fc620af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" May 25 13:06:29.168767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.168756 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3adf5aa3c0fea559adbe132dae0b47e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal\" (UID: \"e3adf5aa3c0fea559adbe132dae0b47e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:29.168829 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.168787 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3adf5aa3c0fea559adbe132dae0b47e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal\" (UID: \"e3adf5aa3c0fea559adbe132dae0b47e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:29.168829 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.168818 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48fb10cd0fb028145d43ec292fc620af-config\") pod \"kube-apiserver-proxy-ip-10-0-135-64.ec2.internal\" (UID: \"48fb10cd0fb028145d43ec292fc620af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" May 25 13:06:29.264046 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.263993 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.322455 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.322417 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:29.327798 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.327784 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" May 25 13:06:29.364252 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.364225 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.464701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.464671 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.565237 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.565192 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.665718 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.665695 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.674142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.674125 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 25 13:06:29.674278 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.674258 2554 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 25 13:06:29.674339 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.674288 2554 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 25 13:06:29.764866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.764845 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 25 13:06:29.767131 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.767115 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.769005 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.768965 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 13:01:28 +0000 UTC" deadline="2027-11-28 22:46:04.326697799 +0000 UTC" May 25 13:06:29.769065 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.769006 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13257h39m34.557695662s" May 25 13:06:29.778260 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.778242 2554 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 25 13:06:29.800956 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.800931 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8rfbh" May 25 13:06:29.809647 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.809630 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8rfbh" May 25 13:06:29.867821 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.867798 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:29.908784 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:29.908647 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3adf5aa3c0fea559adbe132dae0b47e.slice/crio-e82e8acad180187b05728187bbb96893af38ecf96483a603ea056ba7a7d0c9eb WatchSource:0}: Error finding container e82e8acad180187b05728187bbb96893af38ecf96483a603ea056ba7a7d0c9eb: Status 404 returned error can't find the container with id e82e8acad180187b05728187bbb96893af38ecf96483a603ea056ba7a7d0c9eb May 25 13:06:29.910135 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:29.910110 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fb10cd0fb028145d43ec292fc620af.slice/crio-77be5376d7f44ca13778953231d2efe3033fd9c348a2df59ae046df5edeb0a2e WatchSource:0}: Error finding container 77be5376d7f44ca13778953231d2efe3033fd9c348a2df59ae046df5edeb0a2e: Status 404 returned error can't find the container with id 77be5376d7f44ca13778953231d2efe3033fd9c348a2df59ae046df5edeb0a2e May 25 13:06:29.913569 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:29.913530 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 13:06:29.968016 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:29.967994 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:30.068389 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.068370 2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-64.ec2.internal\" not found" May 25 13:06:30.088822 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.088782 2554 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 25 13:06:30.167686 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.167667 2554 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" May 25 13:06:30.179982 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.179967 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 25 13:06:30.182017 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.182005 2554 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" May 25 13:06:30.188701 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.188690 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 25 13:06:30.266409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.266388 2554 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 25 13:06:30.745762 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.745736 2554 apiserver.go:52] "Watching apiserver" May 25 13:06:30.755071 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.755047 2554 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 25 13:06:30.757245 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.757212 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-bhlgr","openshift-network-operator/iptables-alerter-b682t","openshift-ovn-kubernetes/ovnkube-node-4wtp2","kube-system/konnectivity-agent-sq2c5","kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bxl5h","openshift-dns/node-resolver-d5p4k","openshift-image-registry/node-ca-5brhb","openshift-multus/multus-additional-cni-plugins-b7wr8","openshift-multus/multus-mkxpd","openshift-multus/network-metrics-daemon-2228g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal"] May 25 13:06:30.758927 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.758902 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:30.759040 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.758995 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:30.761541 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.761393 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.761541 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.761486 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.762931 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.762853 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:30.763737 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.763717 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 25 13:06:30.763826 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.763804 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 25 13:06:30.764020 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.763984 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 25 13:06:30.764183 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.764167 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 25 13:06:30.764322 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.764302 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9pmml\"" May 25 13:06:30.764398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.764361 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 25 13:06:30.764482 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.764402 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzxcp\"" May 25 13:06:30.765164 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.765059 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.765262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.765195 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.766509 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.766487 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.770169 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.769474 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.771054 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.770984 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 25 13:06:30.771054 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.771034 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hht9v\"" May 25 13:06:30.771766 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.771747 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mkxpd" May 25 13:06:30.772627 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.772606 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 25 13:06:30.772720 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.772702 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 25 13:06:30.772971 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.772950 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 25 13:06:30.773047 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773005 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 25 13:06:30.773107 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773086 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 25 13:06:30.773204 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773188 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 25 13:06:30.773262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773250 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 25 13:06:30.773339 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773321 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 25 13:06:30.773415 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773396 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 25 13:06:30.773500 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773482 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 25 13:06:30.773568 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.772962 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 25 13:06:30.773656 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773637 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4r4qt\"" May 25 13:06:30.773740 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773724 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 25 13:06:30.773867 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773847 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wrvjj\"" May 25 13:06:30.773972 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773962 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 25 13:06:30.774033 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.773993 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cn9g4\"" May 25 13:06:30.774087 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.774074 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwzfv\"" May 25 13:06:30.774462 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.774447 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 25 13:06:30.774925 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.774907 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:30.775015 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.774993 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:30.776787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776403 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 25 13:06:30.776787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776447 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.776787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776516 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 25 13:06:30.776787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776539 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ggcdh\"" May 25 13:06:30.776787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776577 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 25 13:06:30.777064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776814 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-tuned\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776913 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zq2h\" (UniqueName: \"kubernetes.io/projected/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-kube-api-access-9zq2h\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.777064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776930 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 25 13:06:30.777064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776947 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-modprobe-d\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776972 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-tmp\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.776995 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-cni-netd\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777019 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-lib-modules\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777083 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-env-overrides\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777135 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cnibin\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777165 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5clb\" (UniqueName: \"kubernetes.io/projected/e460f6bd-9a46-46d2-870f-f4d203cf05fd-kube-api-access-z5clb\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777187 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-kubelet\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777218 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-run-ovn-kubernetes\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777253 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-systemd-units\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777278 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-etc-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777300 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-cni-bin\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777321 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-serviceca\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.777346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777339 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-ovnkube-script-lib\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777360 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777377 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysconfig\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777397 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysctl-conf\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777418 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777472 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwbs\" (UniqueName: \"kubernetes.io/projected/48876559-9080-4bcd-9ae5-0efdddcd6d7d-kube-api-access-9kwbs\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777509 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-sys\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777534 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-ovn\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777561 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-ovnkube-config\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777593 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f51c159-d778-4d89-a010-f265bc961088-iptables-alerter-script\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777618 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777646 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-systemd\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777673 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-os-release\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777698 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777720 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysctl-d\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.777744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777749 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-run\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777772 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-systemd\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777795 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-system-cni-dir\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777826 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-node-log\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777860 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777888 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e460f6bd-9a46-46d2-870f-f4d203cf05fd-tmp-dir\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777912 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcbd\" (UniqueName: \"kubernetes.io/projected/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-kube-api-access-zmcbd\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777935 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-var-lib-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.777978 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwzz\" (UniqueName: \"kubernetes.io/projected/7492a595-944d-46bb-a1b3-440331bc5331-kube-api-access-5jwzz\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778002 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778023 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e460f6bd-9a46-46d2-870f-f4d203cf05fd-hosts-file\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778046 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-kubernetes\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778066 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-host\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778088 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-log-socket\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778112 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-host\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778152 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8bac7fa-29df-4ca5-8451-ad222a9d671c-agent-certs\") pod \"konnectivity-agent-sq2c5\" (UID: \"c8bac7fa-29df-4ca5-8451-ad222a9d671c\") " pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:30.778398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778176 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65b46\" (UniqueName: \"kubernetes.io/projected/4f51c159-d778-4d89-a010-f265bc961088-kube-api-access-65b46\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.779024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778204 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-run-netns\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.779024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778228 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7492a595-944d-46bb-a1b3-440331bc5331-ovn-node-metrics-cert\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.779024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778274 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8bac7fa-29df-4ca5-8451-ad222a9d671c-konnectivity-ca\") pod \"konnectivity-agent-sq2c5\" (UID: \"c8bac7fa-29df-4ca5-8451-ad222a9d671c\") " pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:30.779024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778305 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f51c159-d778-4d89-a010-f265bc961088-host-slash\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.779024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:30.779024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778356 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-var-lib-kubelet\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.779024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.778379 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-slash\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.780219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.780199 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 25 13:06:30.780309 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.780294 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 25 13:06:30.780804 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.780789 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t6b65\"" May 25 13:06:30.781040 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.781025 2554 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 25 13:06:30.781498 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.781473 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 25 13:06:30.811093 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.811062 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:01:29 +0000 UTC" deadline="2027-12-08 21:27:42.451699011 +0000 UTC" May 25 13:06:30.811191 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.811094 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13496h21m11.640609436s" May 25 13:06:30.869605 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.868785 2554 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 25 13:06:30.879340 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879308 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-run-netns\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.879465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879347 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7492a595-944d-46bb-a1b3-440331bc5331-ovn-node-metrics-cert\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.879465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879373 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8bac7fa-29df-4ca5-8451-ad222a9d671c-konnectivity-ca\") pod \"konnectivity-agent-sq2c5\" (UID: \"c8bac7fa-29df-4ca5-8451-ad222a9d671c\") " pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:30.879465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879399 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f51c159-d778-4d89-a010-f265bc961088-host-slash\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.879465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879441 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:30.879465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879442 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-run-netns\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879468 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-cni-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-cnibin\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879514 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknwn\" (UniqueName: \"kubernetes.io/projected/3655e2cf-b173-414d-8487-626406b3f39d-kube-api-access-wknwn\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879544 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879549 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f51c159-d778-4d89-a010-f265bc961088-host-slash\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879570 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-var-lib-kubelet\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879593 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-slash\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879619 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-cni-bin\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879644 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-multus-certs\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879670 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-tuned\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879702 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9zq2h\" (UniqueName: \"kubernetes.io/projected/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-kube-api-access-9zq2h\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.879723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879705 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-slash\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879786 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879900 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-device-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879932 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-modprobe-d\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879957 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-tmp\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879984 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-cni-netd\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880011 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-netns\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880034 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-lib-modules\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880042 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-modprobe-d\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880065 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8bac7fa-29df-4ca5-8451-ad222a9d671c-konnectivity-ca\") pod \"konnectivity-agent-sq2c5\" (UID: \"c8bac7fa-29df-4ca5-8451-ad222a9d671c\") " pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880079 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-env-overrides\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880093 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-cni-netd\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.879932 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-var-lib-kubelet\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880109 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cnibin\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880170 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z5clb\" (UniqueName: \"kubernetes.io/projected/e460f6bd-9a46-46d2-870f-f4d203cf05fd-kube-api-access-z5clb\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880193 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-lib-modules\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880229 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cnibin\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880233 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-kubelet\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.880294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880194 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-kubelet\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880281 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-run-ovn-kubernetes\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880308 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-k8s-cni-cncf-io\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880357 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-etc-kubernetes\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880380 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-systemd-units\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880404 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-etc-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880450 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7zc\" (UniqueName: \"kubernetes.io/projected/a311b7af-c362-416f-9ddd-f5b2255dc790-kube-api-access-gv7zc\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880461 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-run-ovn-kubernetes\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880476 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-cni-bin\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880489 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-systemd-units\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880475 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-etc-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880501 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-serviceca\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880515 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-cni-bin\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880526 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3655e2cf-b173-414d-8487-626406b3f39d-cni-binary-copy\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880532 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-env-overrides\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880551 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-ovnkube-script-lib\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880576 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880601 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-socket-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880624 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-etc-selinux\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880649 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-cni-multus\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880671 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-hostroot\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880694 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysconfig\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880716 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysctl-conf\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880747 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880775 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwbs\" (UniqueName: \"kubernetes.io/projected/48876559-9080-4bcd-9ae5-0efdddcd6d7d-kube-api-access-9kwbs\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880801 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880813 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysconfig\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880807 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-conf-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880874 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-serviceca\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880929 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-sys\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.880962 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-ovn\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881010 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-ovn\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881012 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-sys\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.881837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881031 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-ovnkube-config\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881069 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881073 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f51c159-d778-4d89-a010-f265bc961088-iptables-alerter-script\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881111 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysctl-conf\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881151 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881213 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3655e2cf-b173-414d-8487-626406b3f39d-multus-daemon-config\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881319 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-systemd\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881346 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-os-release\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881352 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-ovnkube-script-lib\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881384 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881416 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbl4\" (UniqueName: \"kubernetes.io/projected/1e2b6fca-f227-41c5-9d85-6c3b82019d24-kube-api-access-zqbl4\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881446 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-os-release\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881473 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-registration-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881498 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-systemd\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881505 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-socket-dir-parent\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881531 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-kubelet\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881573 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysctl-d\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.882660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881577 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7492a595-944d-46bb-a1b3-440331bc5331-ovnkube-config\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881600 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-run\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881625 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-systemd\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881652 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881653 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-system-cni-dir\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881700 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-run-systemd\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881703 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-system-cni-dir\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881718 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-run\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881723 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-sysctl-d\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881740 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-node-log\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881772 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881783 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-node-log\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881811 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-os-release\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881824 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881864 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e460f6bd-9a46-46d2-870f-f4d203cf05fd-tmp-dir\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881891 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcbd\" (UniqueName: \"kubernetes.io/projected/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-kube-api-access-zmcbd\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881908 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48876559-9080-4bcd-9ae5-0efdddcd6d7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.883313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881917 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-var-lib-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881953 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-var-lib-openvswitch\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881955 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwzz\" (UniqueName: \"kubernetes.io/projected/7492a595-944d-46bb-a1b3-440331bc5331-kube-api-access-5jwzz\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.881992 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882022 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-system-cni-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882052 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f51c159-d778-4d89-a010-f265bc961088-iptables-alerter-script\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882062 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e460f6bd-9a46-46d2-870f-f4d203cf05fd-hosts-file\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882086 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-kubernetes\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882107 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e460f6bd-9a46-46d2-870f-f4d203cf05fd-hosts-file\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882132 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-host\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882147 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-log-socket\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882145 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e460f6bd-9a46-46d2-870f-f4d203cf05fd-tmp-dir\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882188 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-kubernetes\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882196 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48876559-9080-4bcd-9ae5-0efdddcd6d7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882227 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-host\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882234 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7492a595-944d-46bb-a1b3-440331bc5331-log-socket\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882253 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-host\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882260 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8bac7fa-29df-4ca5-8451-ad222a9d671c-agent-certs\") pod \"konnectivity-agent-sq2c5\" (UID: \"c8bac7fa-29df-4ca5-8451-ad222a9d671c\") " pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:30.883916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882278 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-host\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.884525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882287 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-65b46\" (UniqueName: \"kubernetes.io/projected/4f51c159-d778-4d89-a010-f265bc961088-kube-api-access-65b46\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.884525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.882343 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-sys-fs\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.884525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.883576 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-tmp\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.884525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.883730 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7492a595-944d-46bb-a1b3-440331bc5331-ovn-node-metrics-cert\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.884525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.884186 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-etc-tuned\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.884525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.884450 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8bac7fa-29df-4ca5-8451-ad222a9d671c-agent-certs\") pod \"konnectivity-agent-sq2c5\" (UID: \"c8bac7fa-29df-4ca5-8451-ad222a9d671c\") " pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:30.891694 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.891676 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 13:06:30.891797 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.891697 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 13:06:30.891797 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.891710 2554 projected.go:194] Error preparing data for projected volume kube-api-access-h99zl for pod openshift-network-diagnostics/network-check-target-bhlgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:30.891797 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.891792 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl podName:48e5ebb0-8e9f-498d-bb9e-33568f8b2b02 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:31.391762423 +0000 UTC m=+3.070060019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h99zl" (UniqueName: "kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl") pod "network-check-target-bhlgr" (UID: "48e5ebb0-8e9f-498d-bb9e-33568f8b2b02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:30.893107 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.893055 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zq2h\" (UniqueName: \"kubernetes.io/projected/db1bb1ec-d3b1-43f1-b351-de93dee1b1ea-kube-api-access-9zq2h\") pod \"node-ca-5brhb\" (UID: \"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea\") " pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:30.893678 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.893653 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5clb\" (UniqueName: \"kubernetes.io/projected/e460f6bd-9a46-46d2-870f-f4d203cf05fd-kube-api-access-z5clb\") pod \"node-resolver-d5p4k\" (UID: \"e460f6bd-9a46-46d2-870f-f4d203cf05fd\") " pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:30.894212 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.894181 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwbs\" (UniqueName: \"kubernetes.io/projected/48876559-9080-4bcd-9ae5-0efdddcd6d7d-kube-api-access-9kwbs\") pod \"multus-additional-cni-plugins-b7wr8\" (UID: \"48876559-9080-4bcd-9ae5-0efdddcd6d7d\") " pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:30.897798 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.897758 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcbd\" (UniqueName: \"kubernetes.io/projected/cd8ab44b-a4bb-4222-afe1-a46b0b747cfa-kube-api-access-zmcbd\") pod \"tuned-bxl5h\" (UID: \"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa\") " pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:30.898925 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.898872 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65b46\" (UniqueName: \"kubernetes.io/projected/4f51c159-d778-4d89-a010-f265bc961088-kube-api-access-65b46\") pod \"iptables-alerter-b682t\" (UID: \"4f51c159-d778-4d89-a010-f265bc961088\") " pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:30.899712 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.899661 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwzz\" (UniqueName: \"kubernetes.io/projected/7492a595-944d-46bb-a1b3-440331bc5331-kube-api-access-5jwzz\") pod \"ovnkube-node-4wtp2\" (UID: \"7492a595-944d-46bb-a1b3-440331bc5331\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:30.900413 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.900359 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" event={"ID":"e3adf5aa3c0fea559adbe132dae0b47e","Type":"ContainerStarted","Data":"e82e8acad180187b05728187bbb96893af38ecf96483a603ea056ba7a7d0c9eb"} May 25 13:06:30.901413 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.901394 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" event={"ID":"48fb10cd0fb028145d43ec292fc620af","Type":"ContainerStarted","Data":"77be5376d7f44ca13778953231d2efe3033fd9c348a2df59ae046df5edeb0a2e"} May 25 13:06:30.926150 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.926129 2554 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 25 13:06:30.982939 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.982917 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-etc-kubernetes\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983060 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.982948 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7zc\" (UniqueName: \"kubernetes.io/projected/a311b7af-c362-416f-9ddd-f5b2255dc790-kube-api-access-gv7zc\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.983060 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.982971 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3655e2cf-b173-414d-8487-626406b3f39d-cni-binary-copy\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983060 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983028 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-etc-kubernetes\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983065 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-socket-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.983192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983088 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-etc-selinux\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.983192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983115 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-cni-multus\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983158 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-hostroot\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983186 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-conf-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983219 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3655e2cf-b173-414d-8487-626406b3f39d-multus-daemon-config\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983247 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbl4\" (UniqueName: \"kubernetes.io/projected/1e2b6fca-f227-41c5-9d85-6c3b82019d24-kube-api-access-zqbl4\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983268 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-cni-multus\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983279 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-registration-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983322 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-hostroot\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983317 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-socket-dir-parent\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983353 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-conf-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983363 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-kubelet\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983398 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-os-release\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983447 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-system-cni-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983479 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-sys-fs\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983507 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983538 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-cni-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983561 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-cnibin\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983572 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3655e2cf-b173-414d-8487-626406b3f39d-cni-binary-copy\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983584 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wknwn\" (UniqueName: \"kubernetes.io/projected/3655e2cf-b173-414d-8487-626406b3f39d-kube-api-access-wknwn\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983632 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-cni-bin\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983660 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-multus-certs\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983691 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-device-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983719 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-netns\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.983748 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983749 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983772 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-k8s-cni-cncf-io\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983792 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3655e2cf-b173-414d-8487-626406b3f39d-multus-daemon-config\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983860 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983869 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-socket-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983866 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-multus-certs\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983891 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-netns\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983909 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-run-k8s-cni-cncf-io\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983924 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-etc-selinux\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983942 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-registration-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983970 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-system-cni-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.983975 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-kubelet\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.984014 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-socket-dir-parent\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.984015 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-device-dir\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.984052 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-cnibin\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.984067 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-multus-cni-dir\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.984078 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.984116 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-os-release\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.984336 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.984140 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3655e2cf-b173-414d-8487-626406b3f39d-host-var-lib-cni-bin\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.985186 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:30.984147 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:31.484128148 +0000 UTC m=+3.162425776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:30.985186 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.984195 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a311b7af-c362-416f-9ddd-f5b2255dc790-sys-fs\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.991382 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.991340 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknwn\" (UniqueName: \"kubernetes.io/projected/3655e2cf-b173-414d-8487-626406b3f39d-kube-api-access-wknwn\") pod \"multus-mkxpd\" (UID: \"3655e2cf-b173-414d-8487-626406b3f39d\") " pod="openshift-multus/multus-mkxpd" May 25 13:06:30.991743 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.991714 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7zc\" (UniqueName: \"kubernetes.io/projected/a311b7af-c362-416f-9ddd-f5b2255dc790-kube-api-access-gv7zc\") pod \"aws-ebs-csi-driver-node-mz76j\" (UID: \"a311b7af-c362-416f-9ddd-f5b2255dc790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:30.992368 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:30.992338 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbl4\" (UniqueName: \"kubernetes.io/projected/1e2b6fca-f227-41c5-9d85-6c3b82019d24-kube-api-access-zqbl4\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:31.075765 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.075665 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5brhb" May 25 13:06:31.084157 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.084122 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d5p4k" May 25 13:06:31.093728 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.093707 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:31.100270 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.100243 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" May 25 13:06:31.107829 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.107811 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:31.117334 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.117306 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b682t" May 25 13:06:31.123872 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.123853 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" May 25 13:06:31.130376 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.130357 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mkxpd" May 25 13:06:31.136915 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.136896 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" May 25 13:06:31.487517 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.487446 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:31.487517 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.487495 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:31.487701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:31.487525 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:31.487701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:31.487619 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:32.4876038 +0000 UTC m=+4.165901384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:31.487701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:31.487622 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 13:06:31.487701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:31.487638 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 13:06:31.487701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:31.487646 2554 projected.go:194] Error preparing data for projected volume kube-api-access-h99zl for pod openshift-network-diagnostics/network-check-target-bhlgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:31.487701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:31.487696 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl podName:48e5ebb0-8e9f-498d-bb9e-33568f8b2b02 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:32.487685169 +0000 UTC m=+4.165982749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h99zl" (UniqueName: "kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl") pod "network-check-target-bhlgr" (UID: "48e5ebb0-8e9f-498d-bb9e-33568f8b2b02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:31.514888 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:31.514865 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7492a595_944d_46bb_a1b3_440331bc5331.slice/crio-3c7d49c7b6520a9726f66f70db457bcb068fb6bc187b08d2459d8b773e417e2c WatchSource:0}: Error finding container 3c7d49c7b6520a9726f66f70db457bcb068fb6bc187b08d2459d8b773e417e2c: Status 404 returned error can't find the container with id 3c7d49c7b6520a9726f66f70db457bcb068fb6bc187b08d2459d8b773e417e2c May 25 13:06:31.515926 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:31.515897 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1bb1ec_d3b1_43f1_b351_de93dee1b1ea.slice/crio-6a712283c690ed02003331f2a49fa235100cf4fcf2cc0f024c93df0ac6c77988 WatchSource:0}: Error finding container 6a712283c690ed02003331f2a49fa235100cf4fcf2cc0f024c93df0ac6c77988: Status 404 returned error can't find the container with id 6a712283c690ed02003331f2a49fa235100cf4fcf2cc0f024c93df0ac6c77988 May 25 13:06:31.521623 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:31.521333 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8ab44b_a4bb_4222_afe1_a46b0b747cfa.slice/crio-258837f32091b34864c91932e30b5613a69f5dd32aa09ba38b861057ab63772d WatchSource:0}: Error finding container 258837f32091b34864c91932e30b5613a69f5dd32aa09ba38b861057ab63772d: Status 404 returned error can't find the container with id 258837f32091b34864c91932e30b5613a69f5dd32aa09ba38b861057ab63772d May 25 13:06:31.523779 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:31.523660 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bac7fa_29df_4ca5_8451_ad222a9d671c.slice/crio-2fdca1528d1789dc4768037a057fd3dd6f52fcdce2111d8e380f031532b3dcda WatchSource:0}: Error finding container 2fdca1528d1789dc4768037a057fd3dd6f52fcdce2111d8e380f031532b3dcda: Status 404 returned error can't find the container with id 2fdca1528d1789dc4768037a057fd3dd6f52fcdce2111d8e380f031532b3dcda May 25 13:06:31.524174 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:31.524133 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda311b7af_c362_416f_9ddd_f5b2255dc790.slice/crio-8dfae2e768227599462146468ba163a2373d0e2ccb7fa5ee15a13b3d3923c701 WatchSource:0}: Error finding container 8dfae2e768227599462146468ba163a2373d0e2ccb7fa5ee15a13b3d3923c701: Status 404 returned error can't find the container with id 8dfae2e768227599462146468ba163a2373d0e2ccb7fa5ee15a13b3d3923c701 May 25 13:06:31.525954 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:31.525853 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3655e2cf_b173_414d_8487_626406b3f39d.slice/crio-7ee2109ef1fc12a1ea2e861fa179322ef316476a81849e7df7aa7f41ab6cf432 WatchSource:0}: Error finding container 7ee2109ef1fc12a1ea2e861fa179322ef316476a81849e7df7aa7f41ab6cf432: Status 404 returned error can't find the container with id 7ee2109ef1fc12a1ea2e861fa179322ef316476a81849e7df7aa7f41ab6cf432 May 25 13:06:31.528390 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:06:31.527232 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48876559_9080_4bcd_9ae5_0efdddcd6d7d.slice/crio-4c00204a461b72028ab7ea4c758126e5573eab13ee6743c10cd4b71d4afd32d6 WatchSource:0}: Error finding container 4c00204a461b72028ab7ea4c758126e5573eab13ee6743c10cd4b71d4afd32d6: Status 404 returned error can't find the container with id 4c00204a461b72028ab7ea4c758126e5573eab13ee6743c10cd4b71d4afd32d6 May 25 13:06:31.811358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.811323 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:01:29 +0000 UTC" deadline="2027-11-26 17:28:30.368081314 +0000 UTC" May 25 13:06:31.811358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.811350 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13204h21m58.556734398s" May 25 13:06:31.904314 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.904290 2554 generic.go:358] "Generic (PLEG): container finished" podID="e3adf5aa3c0fea559adbe132dae0b47e" containerID="2debf5c8da736c89cfc39f0f57c8e1a053256dff42aa31214c48bc044f9c7c8e" exitCode=0 May 25 13:06:31.904418 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.904357 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" event={"ID":"e3adf5aa3c0fea559adbe132dae0b47e","Type":"ContainerDied","Data":"2debf5c8da736c89cfc39f0f57c8e1a053256dff42aa31214c48bc044f9c7c8e"} May 25 13:06:31.905366 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.905342 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b682t" event={"ID":"4f51c159-d778-4d89-a010-f265bc961088","Type":"ContainerStarted","Data":"3a92248ed5809bea51078837ac4982b7f79143395dfe6d5d172bac215fcd64db"} May 25 13:06:31.906302 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.906280 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerStarted","Data":"4c00204a461b72028ab7ea4c758126e5573eab13ee6743c10cd4b71d4afd32d6"} May 25 13:06:31.907259 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.907216 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" event={"ID":"a311b7af-c362-416f-9ddd-f5b2255dc790","Type":"ContainerStarted","Data":"8dfae2e768227599462146468ba163a2373d0e2ccb7fa5ee15a13b3d3923c701"} May 25 13:06:31.908098 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.908073 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sq2c5" event={"ID":"c8bac7fa-29df-4ca5-8451-ad222a9d671c","Type":"ContainerStarted","Data":"2fdca1528d1789dc4768037a057fd3dd6f52fcdce2111d8e380f031532b3dcda"} May 25 13:06:31.909012 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.908983 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5brhb" event={"ID":"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea","Type":"ContainerStarted","Data":"6a712283c690ed02003331f2a49fa235100cf4fcf2cc0f024c93df0ac6c77988"} May 25 13:06:31.910008 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.909982 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"3c7d49c7b6520a9726f66f70db457bcb068fb6bc187b08d2459d8b773e417e2c"} May 25 13:06:31.911257 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.911237 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" event={"ID":"48fb10cd0fb028145d43ec292fc620af","Type":"ContainerStarted","Data":"27816201a2334523bbc598d06e0d373dc42b2b6336214debc99117d94ebec5ac"} May 25 13:06:31.912229 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.912210 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d5p4k" event={"ID":"e460f6bd-9a46-46d2-870f-f4d203cf05fd","Type":"ContainerStarted","Data":"e1acb824d5cd90fe4b229acd01d0db64ded722d776edd7ca448eb2b92b3e0253"} May 25 13:06:31.913185 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.913166 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mkxpd" event={"ID":"3655e2cf-b173-414d-8487-626406b3f39d","Type":"ContainerStarted","Data":"7ee2109ef1fc12a1ea2e861fa179322ef316476a81849e7df7aa7f41ab6cf432"} May 25 13:06:31.914232 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.914215 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" event={"ID":"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa","Type":"ContainerStarted","Data":"258837f32091b34864c91932e30b5613a69f5dd32aa09ba38b861057ab63772d"} May 25 13:06:31.935261 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:31.935216 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-64.ec2.internal" podStartSLOduration=1.93520501 podStartE2EDuration="1.93520501s" podCreationTimestamp="2026-05-25 13:06:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:31.934660158 +0000 UTC m=+3.612957761" watchObservedRunningTime="2026-05-25 13:06:31.93520501 +0000 UTC m=+3.613502612" May 25 13:06:32.494265 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:32.494184 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:32.494265 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:32.494235 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:32.494507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.494392 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 13:06:32.494507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.494410 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 13:06:32.494507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.494437 2554 projected.go:194] Error preparing data for projected volume kube-api-access-h99zl for pod openshift-network-diagnostics/network-check-target-bhlgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:32.494507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.494495 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl podName:48e5ebb0-8e9f-498d-bb9e-33568f8b2b02 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:34.494477477 +0000 UTC m=+6.172775073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h99zl" (UniqueName: "kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl") pod "network-check-target-bhlgr" (UID: "48e5ebb0-8e9f-498d-bb9e-33568f8b2b02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:32.494895 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.494870 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:32.494961 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.494927 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:34.494911677 +0000 UTC m=+6.173209264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:32.897669 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:32.897591 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:32.898081 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.897726 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:32.898081 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:32.897802 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:32.898081 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:32.897911 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:32.925449 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:32.925380 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" event={"ID":"e3adf5aa3c0fea559adbe132dae0b47e","Type":"ContainerStarted","Data":"ff9ccd72f31e7f97224d4d30d7e00c6dfab897af5df3b6410b1951cd1205ba37"} May 25 13:06:32.949576 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:32.949518 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-64.ec2.internal" podStartSLOduration=2.949502833 podStartE2EDuration="2.949502833s" podCreationTimestamp="2026-05-25 13:06:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:32.948135282 +0000 UTC m=+4.626432877" watchObservedRunningTime="2026-05-25 13:06:32.949502833 +0000 UTC m=+4.627800437" May 25 13:06:34.509468 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:34.509410 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:34.509906 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:34.509491 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:34.509906 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.509634 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 13:06:34.509906 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.509650 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 13:06:34.509906 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.509663 2554 projected.go:194] Error preparing data for projected volume kube-api-access-h99zl for pod openshift-network-diagnostics/network-check-target-bhlgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:34.509906 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.509721 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl podName:48e5ebb0-8e9f-498d-bb9e-33568f8b2b02 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.509703407 +0000 UTC m=+10.188000994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h99zl" (UniqueName: "kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl") pod "network-check-target-bhlgr" (UID: "48e5ebb0-8e9f-498d-bb9e-33568f8b2b02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:34.510174 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.510134 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:34.510231 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.510188 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.510170824 +0000 UTC m=+10.188468405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:34.898297 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:34.897566 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:34.898297 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.897686 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:34.898297 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:34.898101 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:34.898297 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:34.898200 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:36.896419 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:36.896388 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:36.896842 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:36.896415 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:36.896842 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:36.896583 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:36.896842 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:36.896687 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:38.540596 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:38.540522 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:38.540596 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:38.540575 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:38.541230 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.540720 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 13:06:38.541230 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.540744 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 13:06:38.541230 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.540749 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:38.541230 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.540758 2554 projected.go:194] Error preparing data for projected volume kube-api-access-h99zl for pod openshift-network-diagnostics/network-check-target-bhlgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:38.541230 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.540817 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:46.540795083 +0000 UTC m=+18.219092683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:38.541230 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.540838 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl podName:48e5ebb0-8e9f-498d-bb9e-33568f8b2b02 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:46.540827795 +0000 UTC m=+18.219125376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h99zl" (UniqueName: "kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl") pod "network-check-target-bhlgr" (UID: "48e5ebb0-8e9f-498d-bb9e-33568f8b2b02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:38.897842 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:38.897770 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:38.897984 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.897936 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:38.898049 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:38.898029 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:38.898285 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:38.898230 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:40.896751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:40.896719 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:40.897189 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:40.896733 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:40.897189 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:40.896844 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:40.897189 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:40.896898 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:42.897201 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:42.897026 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:42.897569 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:42.897076 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:42.897569 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:42.897278 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:42.897569 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:42.897365 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:44.897132 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:44.897101 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:44.897132 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:44.897121 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:44.897624 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:44.897219 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:44.897624 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:44.897342 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:46.606082 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:46.606050 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:46.606082 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:46.606088 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:46.606546 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.606212 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:46.606546 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.606226 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 13:06:46.606546 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.606246 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 13:06:46.606546 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.606255 2554 projected.go:194] Error preparing data for projected volume kube-api-access-h99zl for pod openshift-network-diagnostics/network-check-target-bhlgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:46.606546 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.606282 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:02.606260629 +0000 UTC m=+34.284558213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:06:46.606546 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.606301 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl podName:48e5ebb0-8e9f-498d-bb9e-33568f8b2b02 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:02.606290719 +0000 UTC m=+34.284588302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h99zl" (UniqueName: "kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl") pod "network-check-target-bhlgr" (UID: "48e5ebb0-8e9f-498d-bb9e-33568f8b2b02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:06:46.897077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:46.897002 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:46.897238 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:46.897002 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:46.897238 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.897125 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:46.897238 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:46.897220 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:48.897284 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:48.897254 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:48.897671 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:48.897340 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:48.897671 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:48.897446 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:48.897671 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:48.897532 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:49.954348 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.954126 2554 generic.go:358] "Generic (PLEG): container finished" podID="48876559-9080-4bcd-9ae5-0efdddcd6d7d" containerID="ea4676c81892291ad6fb068fe4fb522faf701b4e00d8b175396411a966dc9f22" exitCode=0 May 25 13:06:49.955064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.954216 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerDied","Data":"ea4676c81892291ad6fb068fe4fb522faf701b4e00d8b175396411a966dc9f22"} May 25 13:06:49.955843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.955603 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" event={"ID":"a311b7af-c362-416f-9ddd-f5b2255dc790","Type":"ContainerStarted","Data":"a21588e5d4e2ba7334739fb7db21fd139f2e186de28175ed5f2c9e95fef5f58f"} May 25 13:06:49.956857 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.956835 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sq2c5" event={"ID":"c8bac7fa-29df-4ca5-8451-ad222a9d671c","Type":"ContainerStarted","Data":"5ba964236be63a28b9fd5df6ca90796511feae0cef4f182fbc22e131f6caad12"} May 25 13:06:49.958134 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.958097 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5brhb" event={"ID":"db1bb1ec-d3b1-43f1-b351-de93dee1b1ea","Type":"ContainerStarted","Data":"220a073a50f86a0ba108510a02da51a8013291aa43d4ad47d544d24675019851"} May 25 13:06:49.960681 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.960655 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"37a8c8f96bda860a4f3dfe60d27b14595088d44011c28f18ef6954931ac2043b"} May 25 13:06:49.960681 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.960682 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"32a54c199ee3e9294dab6a7769442ae38e463a146822c96e177dafdcabb98aae"} May 25 13:06:49.960829 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.960693 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"f818c2f52bf7f0fc3fb1023cc47f282de93e88a7e50b271e24f6dabfd1b78da0"} May 25 13:06:49.960829 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.960704 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"e582c2a1776f0d30640e756b9760e5b25f457a3283cb11d345593c9965adc2fc"} May 25 13:06:49.960829 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.960713 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"b422f82699b6bc5de980def782b3cc17a890324160de5d73ea3b8b50878c4aa6"} May 25 13:06:49.960829 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.960721 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"a2ebe4e802f368e519ab7cdbe54f914dcdab7bb9bd1d4598997bbe0f8d6b24fa"} May 25 13:06:49.962267 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.962248 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d5p4k" event={"ID":"e460f6bd-9a46-46d2-870f-f4d203cf05fd","Type":"ContainerStarted","Data":"df30282a9303fcd95b5c74f6cb140ddc30f5ba408077538f0fbbe3239e140c43"} May 25 13:06:49.963471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.963451 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mkxpd" event={"ID":"3655e2cf-b173-414d-8487-626406b3f39d","Type":"ContainerStarted","Data":"e4ca303bcc39240913a7cd2227405079c8de38b1117a4f8fad9539de6581d228"} May 25 13:06:49.964517 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.964496 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" event={"ID":"cd8ab44b-a4bb-4222-afe1-a46b0b747cfa","Type":"ContainerStarted","Data":"6f58fd93da8142c95eaa6fde1cdad44d114df343330f0e41c29c0a1ebc53d2e5"} May 25 13:06:49.990632 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:49.990596 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-d5p4k" podStartSLOduration=3.186348554 podStartE2EDuration="20.990585214s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.531016156 +0000 UTC m=+3.209313741" lastFinishedPulling="2026-05-25 13:06:49.335252809 +0000 UTC m=+21.013550401" observedRunningTime="2026-05-25 13:06:49.990357521 +0000 UTC m=+21.668655123" watchObservedRunningTime="2026-05-25 13:06:49.990585214 +0000 UTC m=+21.668882815" May 25 13:06:50.026991 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.026956 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bxl5h" podStartSLOduration=3.2119766 podStartE2EDuration="21.026946962s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.524713748 +0000 UTC m=+3.203011338" lastFinishedPulling="2026-05-25 13:06:49.339684115 +0000 UTC m=+21.017981700" observedRunningTime="2026-05-25 13:06:50.009952819 +0000 UTC m=+21.688250432" watchObservedRunningTime="2026-05-25 13:06:50.026946962 +0000 UTC m=+21.705244564" May 25 13:06:50.027084 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.027063 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5brhb" podStartSLOduration=12.362870672 podStartE2EDuration="22.027058914s" podCreationTimestamp="2026-05-25 13:06:28 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.520667478 +0000 UTC m=+3.198965067" lastFinishedPulling="2026-05-25 13:06:41.184855715 +0000 UTC m=+12.863153309" observedRunningTime="2026-05-25 13:06:50.026590507 +0000 UTC m=+21.704888110" watchObservedRunningTime="2026-05-25 13:06:50.027058914 +0000 UTC m=+21.705356515" May 25 13:06:50.043245 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.043062 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sq2c5" podStartSLOduration=3.663860124 podStartE2EDuration="21.043052215s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.527573977 +0000 UTC m=+3.205871568" lastFinishedPulling="2026-05-25 13:06:48.906766074 +0000 UTC m=+20.585063659" observedRunningTime="2026-05-25 13:06:50.042685308 +0000 UTC m=+21.720982910" watchObservedRunningTime="2026-05-25 13:06:50.043052215 +0000 UTC m=+21.721349815" May 25 13:06:50.506358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.506336 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 25 13:06:50.843665 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.843543 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T13:06:50.506353033Z","UUID":"dbeaf565-c8fd-4b6b-9dd4-07fcfb429fa5","Handler":null,"Name":"","Endpoint":""} May 25 13:06:50.845313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.845290 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 25 13:06:50.845496 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.845321 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 25 13:06:50.896505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.896438 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:50.896505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.896465 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:50.896659 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:50.896551 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:50.896729 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:50.896705 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:50.967363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.967334 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b682t" event={"ID":"4f51c159-d778-4d89-a010-f265bc961088","Type":"ContainerStarted","Data":"7b36ce0569f293cc79eb9a499a0de754563d480a69a2e3f7dfede47b6fc5911c"} May 25 13:06:50.968861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.968834 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" event={"ID":"a311b7af-c362-416f-9ddd-f5b2255dc790","Type":"ContainerStarted","Data":"cb506ef43c0404664ca5bbc3c7f6ade65016107f2b4a8cc7bce337ccea7b0100"} May 25 13:06:50.982967 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.982934 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b682t" podStartSLOduration=4.179131167 podStartE2EDuration="21.982923431s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.531496217 +0000 UTC m=+3.209793805" lastFinishedPulling="2026-05-25 13:06:49.335288476 +0000 UTC m=+21.013586069" observedRunningTime="2026-05-25 13:06:50.982855083 +0000 UTC m=+22.661152696" watchObservedRunningTime="2026-05-25 13:06:50.982923431 +0000 UTC m=+22.661221030" May 25 13:06:50.983337 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:50.983317 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-mkxpd" podStartSLOduration=4.151853241 podStartE2EDuration="21.983312114s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.529366311 +0000 UTC m=+3.207663906" lastFinishedPulling="2026-05-25 13:06:49.360825196 +0000 UTC m=+21.039122779" observedRunningTime="2026-05-25 13:06:50.066815153 +0000 UTC m=+21.745112757" watchObservedRunningTime="2026-05-25 13:06:50.983312114 +0000 UTC m=+22.661609718" May 25 13:06:51.972697 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:51.972652 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" event={"ID":"a311b7af-c362-416f-9ddd-f5b2255dc790","Type":"ContainerStarted","Data":"c0cfa7df37a04d47334ea411570af31eb2e379f945391c7d86d19a01c8ceabbd"} May 25 13:06:51.975506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:51.975480 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"cee2eaceb2097e8ec44d6bb71f5583d932e88e3b7cad1d8477e1d308af704a30"} May 25 13:06:51.989553 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:51.989513 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mz76j" podStartSLOduration=2.9767401639999997 podStartE2EDuration="22.989500413s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.527878231 +0000 UTC m=+3.206175824" lastFinishedPulling="2026-05-25 13:06:51.540638493 +0000 UTC m=+23.218936073" observedRunningTime="2026-05-25 13:06:51.989343072 +0000 UTC m=+23.667640673" watchObservedRunningTime="2026-05-25 13:06:51.989500413 +0000 UTC m=+23.667798015" May 25 13:06:52.898789 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:52.898763 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:52.898921 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:52.898764 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:52.898921 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:52.898859 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:52.898995 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:52.898932 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:52.948742 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:52.948715 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:52.949257 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:52.949237 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:54.848493 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:54.848455 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:54.849112 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:54.848566 2554 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" May 25 13:06:54.849362 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:54.849343 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sq2c5" May 25 13:06:54.899552 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:54.899533 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:54.899671 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:54.899628 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:54.899730 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:54.899685 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:54.899815 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:54.899796 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:55.983873 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:55.983657 2554 generic.go:358] "Generic (PLEG): container finished" podID="48876559-9080-4bcd-9ae5-0efdddcd6d7d" containerID="908912531d93cb15c6ceb8a8d3a474e6d57291a20179e562df51fa786bed0229" exitCode=0 May 25 13:06:55.984214 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:55.983743 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerDied","Data":"908912531d93cb15c6ceb8a8d3a474e6d57291a20179e562df51fa786bed0229"} May 25 13:06:55.987565 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:55.987542 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" event={"ID":"7492a595-944d-46bb-a1b3-440331bc5331","Type":"ContainerStarted","Data":"79c2443ea4d8bcc7ee569eb5f9be8d1025fd4be88d0dc64f16af0d428416f8a5"} May 25 13:06:55.987951 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:55.987930 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:55.988053 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:55.987961 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:55.988053 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:55.987972 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:56.002965 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:56.002941 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:56.003066 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:56.003038 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:06:56.038384 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:56.038342 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" podStartSLOduration=9.15032022 podStartE2EDuration="27.038332392s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.517325462 +0000 UTC m=+3.195623045" lastFinishedPulling="2026-05-25 13:06:49.405337624 +0000 UTC m=+21.083635217" observedRunningTime="2026-05-25 13:06:56.038260723 +0000 UTC m=+27.716558328" watchObservedRunningTime="2026-05-25 13:06:56.038332392 +0000 UTC m=+27.716629994" May 25 13:06:56.896823 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:56.896792 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:56.897056 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:56.896896 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:56.897056 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:56.896923 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:56.897056 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:56.896966 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:57.183743 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:57.183555 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2228g"] May 25 13:06:57.184146 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:57.183775 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:57.184146 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:57.183865 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:57.186546 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:57.186523 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bhlgr"] May 25 13:06:57.186637 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:57.186608 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:57.186718 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:57.186697 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:57.993179 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:57.993146 2554 generic.go:358] "Generic (PLEG): container finished" podID="48876559-9080-4bcd-9ae5-0efdddcd6d7d" containerID="b7ebe7e2b555e53dc076c0a9c77f8b03b6634bbc40c21f066df7c1b70b4f86c3" exitCode=0 May 25 13:06:57.993362 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:57.993241 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerDied","Data":"b7ebe7e2b555e53dc076c0a9c77f8b03b6634bbc40c21f066df7c1b70b4f86c3"} May 25 13:06:58.896997 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:58.896968 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:06:58.897492 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:58.897041 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:06:58.897492 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:58.897124 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:06:58.897492 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:06:58.897212 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:06:59.997343 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:59.997311 2554 generic.go:358] "Generic (PLEG): container finished" podID="48876559-9080-4bcd-9ae5-0efdddcd6d7d" containerID="01f8dd5d37804c20566dea655b1003c27e8629cc90a52ae6030f36e9019606bf" exitCode=0 May 25 13:06:59.997814 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:06:59.997351 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerDied","Data":"01f8dd5d37804c20566dea655b1003c27e8629cc90a52ae6030f36e9019606bf"} May 25 13:07:00.896934 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:00.896904 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:07:00.897079 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:00.896997 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bhlgr" podUID="48e5ebb0-8e9f-498d-bb9e-33568f8b2b02" May 25 13:07:00.897134 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:00.897083 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:07:00.897217 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:00.897195 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2228g" podUID="1e2b6fca-f227-41c5-9d85-6c3b82019d24" May 25 13:07:02.113666 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.113587 2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-64.ec2.internal" event="NodeReady" May 25 13:07:02.114092 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.113740 2554 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 25 13:07:02.159781 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.159751 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h8xqz"] May 25 13:07:02.177500 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.177476 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-64d4d"] May 25 13:07:02.177655 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.177637 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.182312 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.182283 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 25 13:07:02.182460 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.182440 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dnmsw\"" May 25 13:07:02.182672 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.182651 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 25 13:07:02.193566 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.193493 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h8xqz"] May 25 13:07:02.193645 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.193589 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-64d4d"] May 25 13:07:02.193645 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.193597 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:02.195991 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.195909 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 25 13:07:02.195991 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.195972 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 25 13:07:02.196198 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.196178 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 25 13:07:02.196884 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.196869 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6hqsn\"" May 25 13:07:02.328606 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.328576 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:02.328754 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.328608 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.328754 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.328638 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrtf\" (UniqueName: \"kubernetes.io/projected/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-kube-api-access-hvrtf\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:02.328754 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.328701 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-config-volume\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.328754 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.328721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ggw\" (UniqueName: \"kubernetes.io/projected/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-kube-api-access-n6ggw\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.328906 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.328790 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-tmp-dir\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.429945 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.429879 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrtf\" (UniqueName: \"kubernetes.io/projected/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-kube-api-access-hvrtf\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:02.429945 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.429915 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-config-volume\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.429945 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.429936 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ggw\" (UniqueName: \"kubernetes.io/projected/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-kube-api-access-n6ggw\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.430196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.429956 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-tmp-dir\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.430196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.430003 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:02.430196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.430018 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.430196 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.430101 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:07:02.430196 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.430124 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:07:02.430196 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.430155 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:02.930138287 +0000 UTC m=+34.608435881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:07:02.430196 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.430179 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:02.930168066 +0000 UTC m=+34.608465660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:07:02.430420 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.430353 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-tmp-dir\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.430516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.430498 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-config-volume\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.444480 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.444326 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ggw\" (UniqueName: \"kubernetes.io/projected/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-kube-api-access-n6ggw\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.444591 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.444385 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrtf\" (UniqueName: \"kubernetes.io/projected/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-kube-api-access-hvrtf\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:02.631252 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.631217 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:07:02.631417 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.631262 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:07:02.631417 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.631363 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:07:02.631536 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.631367 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 25 13:07:02.631536 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.631467 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:34.631445521 +0000 UTC m=+66.309743104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : object "openshift-multus"/"metrics-daemon-secret" not registered May 25 13:07:02.631536 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.631470 2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 25 13:07:02.631536 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.631482 2554 projected.go:194] Error preparing data for projected volume kube-api-access-h99zl for pod openshift-network-diagnostics/network-check-target-bhlgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:07:02.631536 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.631526 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl podName:48e5ebb0-8e9f-498d-bb9e-33568f8b2b02 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:34.631509298 +0000 UTC m=+66.309806884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h99zl" (UniqueName: "kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl") pod "network-check-target-bhlgr" (UID: "48e5ebb0-8e9f-498d-bb9e-33568f8b2b02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 25 13:07:02.897134 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.897103 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:07:02.897314 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.897292 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:07:02.901542 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.901522 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 25 13:07:02.901542 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.901532 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 25 13:07:02.901724 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.901577 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-stbll\"" May 25 13:07:02.902319 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.902298 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xxkxl\"" May 25 13:07:02.902319 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.902316 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 25 13:07:02.933712 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.933692 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:02.933802 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:02.933717 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:02.933848 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.933823 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:07:02.933883 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.933864 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:07:02.933883 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.933881 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:03.933866552 +0000 UTC m=+35.612164131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:07:02.933959 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:02.933898 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:03.933886982 +0000 UTC m=+35.612184563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:07:03.941633 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:03.941603 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:03.942192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:03.941638 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:03.942192 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:03.941776 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:07:03.942192 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:03.941807 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:07:03.942192 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:03.941856 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:05.941836138 +0000 UTC m=+37.620133735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:07:03.942192 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:03.941877 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:05.941867836 +0000 UTC m=+37.620165422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:07:05.955092 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:05.955031 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:05.955092 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:05.955089 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:05.955655 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:05.955151 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:07:05.955655 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:05.955237 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:09.955213208 +0000 UTC m=+41.633510805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:07:05.955655 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:05.955240 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:07:05.955655 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:05.955300 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:09.955281479 +0000 UTC m=+41.633579078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:07:08.014296 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:08.014211 2554 generic.go:358] "Generic (PLEG): container finished" podID="48876559-9080-4bcd-9ae5-0efdddcd6d7d" containerID="eacbfd09fe02935407b7427f9f71ad72006b73428ffa9dd6831b56d1ca2f1426" exitCode=0 May 25 13:07:08.014296 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:08.014277 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerDied","Data":"eacbfd09fe02935407b7427f9f71ad72006b73428ffa9dd6831b56d1ca2f1426"} May 25 13:07:09.018503 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:09.018472 2554 generic.go:358] "Generic (PLEG): container finished" podID="48876559-9080-4bcd-9ae5-0efdddcd6d7d" containerID="517014e08bd6a9cbb2d947418c4ccb085322f9599d51e63e0285113693d09ec5" exitCode=0 May 25 13:07:09.018900 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:09.018510 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerDied","Data":"517014e08bd6a9cbb2d947418c4ccb085322f9599d51e63e0285113693d09ec5"} May 25 13:07:09.985565 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:09.985527 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:09.985754 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:09.985571 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:09.985754 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:09.985688 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:07:09.985874 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:09.985757 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:17.985739324 +0000 UTC m=+49.664036905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:07:09.985874 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:09.985693 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:07:09.985874 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:09.985864 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:17.985844767 +0000 UTC m=+49.664142350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:07:10.024145 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:10.024115 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" event={"ID":"48876559-9080-4bcd-9ae5-0efdddcd6d7d","Type":"ContainerStarted","Data":"7b49c5a94f74ee9dc5384b79a1ab58d93bd895db59599f1a55eaee274d6eda81"} May 25 13:07:10.051693 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:10.051550 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b7wr8" podStartSLOduration=5.237253478 podStartE2EDuration="41.051535249s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:06:31.529722333 +0000 UTC m=+3.208019916" lastFinishedPulling="2026-05-25 13:07:07.344004106 +0000 UTC m=+39.022301687" observedRunningTime="2026-05-25 13:07:10.050137404 +0000 UTC m=+41.728435006" watchObservedRunningTime="2026-05-25 13:07:10.051535249 +0000 UTC m=+41.729832853" May 25 13:07:18.036540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:18.036506 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:18.036540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:18.036542 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:18.036956 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:18.036634 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:07:18.036956 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:18.036638 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:07:18.036956 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:18.036682 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:34.036668669 +0000 UTC m=+65.714966249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:07:18.036956 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:18.036694 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:34.036688097 +0000 UTC m=+65.714985676 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:07:28.012389 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:28.012355 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wtp2" May 25 13:07:34.037969 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.037933 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:07:34.037969 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.037969 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:07:34.038507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:34.038063 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:07:34.038507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:34.038066 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:07:34.038507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:34.038136 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:06.038117545 +0000 UTC m=+97.716415142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:07:34.038507 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:34.038153 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:06.038145117 +0000 UTC m=+97.716442696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:07:34.642383 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.642340 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:07:34.642383 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.642386 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:07:34.645039 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.645018 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 25 13:07:34.645084 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.645048 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 25 13:07:34.652911 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:34.652892 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 25 13:07:34.652959 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:07:34.652945 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs podName:1e2b6fca-f227-41c5-9d85-6c3b82019d24 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:38.652930976 +0000 UTC m=+130.331228561 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs") pod "network-metrics-daemon-2228g" (UID: "1e2b6fca-f227-41c5-9d85-6c3b82019d24") : secret "metrics-daemon-secret" not found May 25 13:07:34.654908 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.654894 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 25 13:07:34.667007 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.666978 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h99zl\" (UniqueName: \"kubernetes.io/projected/48e5ebb0-8e9f-498d-bb9e-33568f8b2b02-kube-api-access-h99zl\") pod \"network-check-target-bhlgr\" (UID: \"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02\") " pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:07:34.709137 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.709117 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-stbll\"" May 25 13:07:34.717039 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.717017 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:07:34.840394 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:34.840359 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bhlgr"] May 25 13:07:34.843652 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:07:34.843617 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e5ebb0_8e9f_498d_bb9e_33568f8b2b02.slice/crio-65a2029a2722df1419d50c33422bcd7f8aa2e747a719b374ceefcbba84f4ab7c WatchSource:0}: Error finding container 65a2029a2722df1419d50c33422bcd7f8aa2e747a719b374ceefcbba84f4ab7c: Status 404 returned error can't find the container with id 65a2029a2722df1419d50c33422bcd7f8aa2e747a719b374ceefcbba84f4ab7c May 25 13:07:35.081926 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:35.081901 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bhlgr" event={"ID":"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02","Type":"ContainerStarted","Data":"65a2029a2722df1419d50c33422bcd7f8aa2e747a719b374ceefcbba84f4ab7c"} May 25 13:07:38.088344 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:38.088314 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bhlgr" event={"ID":"48e5ebb0-8e9f-498d-bb9e-33568f8b2b02","Type":"ContainerStarted","Data":"43ed97a6cb23dae0e3e1a8a3a21dc7a9fd2813f460fd0c61a71f530138df50d4"} May 25 13:07:38.088640 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:38.088446 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:07:38.103146 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:07:38.103100 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bhlgr" podStartSLOduration=67.082173345 podStartE2EDuration="1m10.103085847s" podCreationTimestamp="2026-05-25 13:06:28 +0000 UTC" firstStartedPulling="2026-05-25 13:07:34.845443531 +0000 UTC m=+66.523741114" lastFinishedPulling="2026-05-25 13:07:37.866356033 +0000 UTC m=+69.544653616" observedRunningTime="2026-05-25 13:07:38.102571374 +0000 UTC m=+69.780868988" watchObservedRunningTime="2026-05-25 13:07:38.103085847 +0000 UTC m=+69.781383450" May 25 13:08:02.430947 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.430914 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-vt6mr"] May 25 13:08:02.433163 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.433147 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.435522 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.435496 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 25 13:08:02.435621 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.435550 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" May 25 13:08:02.435621 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.435601 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 25 13:08:02.435709 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.435601 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" May 25 13:08:02.436718 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.436702 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cxbz7\"" May 25 13:08:02.440885 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.440867 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" May 25 13:08:02.448125 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.448103 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-vt6mr"] May 25 13:08:02.528533 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.528507 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d027ad-c810-423a-98fa-54547bad1795-service-ca-bundle\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.528533 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.528534 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d027ad-c810-423a-98fa-54547bad1795-tmp\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.528706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.528570 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d027ad-c810-423a-98fa-54547bad1795-snapshots\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.528706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.528612 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d027ad-c810-423a-98fa-54547bad1795-serving-cert\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.528706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.528651 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d027ad-c810-423a-98fa-54547bad1795-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.528706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.528685 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29cqm\" (UniqueName: \"kubernetes.io/projected/69d027ad-c810-423a-98fa-54547bad1795-kube-api-access-29cqm\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.535367 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.535339 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l"] May 25 13:08:02.537058 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.537042 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.540716 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.540700 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" May 25 13:08:02.540843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.540827 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" May 25 13:08:02.541061 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.541044 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" May 25 13:08:02.541191 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.541131 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" May 25 13:08:02.541332 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.541312 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hqqvf\"" May 25 13:08:02.542109 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.542084 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d"] May 25 13:08:02.543954 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.543934 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q"] May 25 13:08:02.544093 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.544077 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.545579 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.545561 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-jfbcj"] May 25 13:08:02.545715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.545698 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:02.547052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.547039 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.557286 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.557269 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" May 25 13:08:02.557379 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.557348 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 25 13:08:02.557379 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.557354 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" May 25 13:08:02.557516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.557407 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wmwhg\"" May 25 13:08:02.560540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.560522 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" May 25 13:08:02.560888 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.560865 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 25 13:08:02.567267 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567249 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" May 25 13:08:02.567355 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567280 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" May 25 13:08:02.567414 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567398 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" May 25 13:08:02.567414 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567409 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" May 25 13:08:02.567530 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567457 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4qv64\"" May 25 13:08:02.567743 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567727 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-j4lw5\"" May 25 13:08:02.567821 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567805 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" May 25 13:08:02.567873 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.567847 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" May 25 13:08:02.568409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.568389 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf"] May 25 13:08:02.570687 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.570670 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l"] May 25 13:08:02.570816 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.570800 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" May 25 13:08:02.574148 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.574129 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-jfbcj"] May 25 13:08:02.575568 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.575546 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" May 25 13:08:02.576018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.575978 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" May 25 13:08:02.576121 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.576093 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xf7md\"" May 25 13:08:02.578563 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.578545 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" May 25 13:08:02.591248 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.591231 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf"] May 25 13:08:02.598145 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.598128 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d"] May 25 13:08:02.629548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629523 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d027ad-c810-423a-98fa-54547bad1795-serving-cert\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.629641 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629554 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d027ad-c810-423a-98fa-54547bad1795-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.629641 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629579 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29cqm\" (UniqueName: \"kubernetes.io/projected/69d027ad-c810-423a-98fa-54547bad1795-kube-api-access-29cqm\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.629641 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629604 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.629641 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629631 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvqn\" (UniqueName: \"kubernetes.io/projected/d5a29851-cc12-4af0-93df-a15c86f0ee86-kube-api-access-vwvqn\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.629789 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629655 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/077160a2-2714-4cf9-961e-4169dc361eaa-config\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.629789 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629709 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/077160a2-2714-4cf9-961e-4169dc361eaa-serving-cert\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.629789 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629755 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/077160a2-2714-4cf9-961e-4169dc361eaa-trusted-ca\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.629908 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629786 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksst\" (UniqueName: \"kubernetes.io/projected/97a552c2-8f68-4558-85d3-300397fe201b-kube-api-access-mksst\") pod \"volume-data-source-validator-5cf8c89d64-8ngrf\" (UID: \"97a552c2-8f68-4558-85d3-300397fe201b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" May 25 13:08:02.629908 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629861 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d027ad-c810-423a-98fa-54547bad1795-service-ca-bundle\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.629908 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629889 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6aeb4909-83b4-4b8a-ac96-d20a74970d44-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.630043 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629918 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d027ad-c810-423a-98fa-54547bad1795-tmp\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.630043 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629942 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzgk\" (UniqueName: \"kubernetes.io/projected/6aeb4909-83b4-4b8a-ac96-d20a74970d44-kube-api-access-rqzgk\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.630043 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:02.630043 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.629978 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5a29851-cc12-4af0-93df-a15c86f0ee86-serving-cert\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.630043 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630000 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5a29851-cc12-4af0-93df-a15c86f0ee86-config\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.630262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630041 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2lm\" (UniqueName: \"kubernetes.io/projected/077160a2-2714-4cf9-961e-4169dc361eaa-kube-api-access-xz2lm\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.630262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630075 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpr7\" (UniqueName: \"kubernetes.io/projected/3df04ad1-0e90-459d-b66b-a391ef478d0f-kube-api-access-tkpr7\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:02.630262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630169 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d027ad-c810-423a-98fa-54547bad1795-snapshots\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.630262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630247 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d027ad-c810-423a-98fa-54547bad1795-tmp\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.630384 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630335 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d027ad-c810-423a-98fa-54547bad1795-service-ca-bundle\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.630417 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630394 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d027ad-c810-423a-98fa-54547bad1795-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.630631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.630616 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69d027ad-c810-423a-98fa-54547bad1795-snapshots\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.632008 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.631989 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69d027ad-c810-423a-98fa-54547bad1795-serving-cert\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.651957 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.651934 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29cqm\" (UniqueName: \"kubernetes.io/projected/69d027ad-c810-423a-98fa-54547bad1795-kube-api-access-29cqm\") pod \"insights-operator-78bdfb888c-vt6mr\" (UID: \"69d027ad-c810-423a-98fa-54547bad1795\") " pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.663227 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.663208 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r"] May 25 13:08:02.665404 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.665388 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.668079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.668053 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" May 25 13:08:02.668079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.668070 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" May 25 13:08:02.668223 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.668110 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6h5zq\"" May 25 13:08:02.668223 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.668070 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" May 25 13:08:02.668948 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.668933 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" May 25 13:08:02.679809 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.679780 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r"] May 25 13:08:02.680516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.680499 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q"] May 25 13:08:02.731471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731383 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6aeb4909-83b4-4b8a-ac96-d20a74970d44-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.731471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731413 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzgk\" (UniqueName: \"kubernetes.io/projected/6aeb4909-83b4-4b8a-ac96-d20a74970d44-kube-api-access-rqzgk\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.731471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731448 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:02.731471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731467 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5a29851-cc12-4af0-93df-a15c86f0ee86-serving-cert\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.731781 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731484 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5a29851-cc12-4af0-93df-a15c86f0ee86-config\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.731781 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:02.731552 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 25 13:08:02.731781 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731609 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2lm\" (UniqueName: \"kubernetes.io/projected/077160a2-2714-4cf9-961e-4169dc361eaa-kube-api-access-xz2lm\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.731781 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:02.731616 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls podName:3df04ad1-0e90-459d-b66b-a391ef478d0f nodeName:}" failed. No retries permitted until 2026-05-25 13:08:03.231593921 +0000 UTC m=+94.909891517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls") pod "cluster-samples-operator-697587599-v7h8q" (UID: "3df04ad1-0e90-459d-b66b-a391ef478d0f") : secret "samples-operator-tls" not found May 25 13:08:02.731781 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731654 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpr7\" (UniqueName: \"kubernetes.io/projected/3df04ad1-0e90-459d-b66b-a391ef478d0f-kube-api-access-tkpr7\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:02.731781 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731707 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/720a0198-112a-4272-b799-e51dbaf56832-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731790 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731821 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvqn\" (UniqueName: \"kubernetes.io/projected/d5a29851-cc12-4af0-93df-a15c86f0ee86-kube-api-access-vwvqn\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731846 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/077160a2-2714-4cf9-961e-4169dc361eaa-config\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:02.731907 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:02.731965 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls podName:6aeb4909-83b4-4b8a-ac96-d20a74970d44 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:03.231949056 +0000 UTC m=+94.910246651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-zb92d" (UID: "6aeb4909-83b4-4b8a-ac96-d20a74970d44") : secret "cluster-monitoring-operator-tls" not found May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.731986 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/077160a2-2714-4cf9-961e-4169dc361eaa-serving-cert\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732017 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/720a0198-112a-4272-b799-e51dbaf56832-config\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6gs\" (UniqueName: \"kubernetes.io/projected/720a0198-112a-4272-b799-e51dbaf56832-kube-api-access-qx6gs\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.732077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732078 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/077160a2-2714-4cf9-961e-4169dc361eaa-trusted-ca\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.732622 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732081 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5a29851-cc12-4af0-93df-a15c86f0ee86-config\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.732622 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732112 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mksst\" (UniqueName: \"kubernetes.io/projected/97a552c2-8f68-4558-85d3-300397fe201b-kube-api-access-mksst\") pod \"volume-data-source-validator-5cf8c89d64-8ngrf\" (UID: \"97a552c2-8f68-4558-85d3-300397fe201b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" May 25 13:08:02.732622 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732195 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6aeb4909-83b4-4b8a-ac96-d20a74970d44-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.732742 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732642 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/077160a2-2714-4cf9-961e-4169dc361eaa-config\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.732823 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.732803 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/077160a2-2714-4cf9-961e-4169dc361eaa-trusted-ca\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.733875 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.733855 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5a29851-cc12-4af0-93df-a15c86f0ee86-serving-cert\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.734281 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.734261 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/077160a2-2714-4cf9-961e-4169dc361eaa-serving-cert\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.742368 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.742351 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" May 25 13:08:02.747751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.747730 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpr7\" (UniqueName: \"kubernetes.io/projected/3df04ad1-0e90-459d-b66b-a391ef478d0f-kube-api-access-tkpr7\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:02.747931 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.747910 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvqn\" (UniqueName: \"kubernetes.io/projected/d5a29851-cc12-4af0-93df-a15c86f0ee86-kube-api-access-vwvqn\") pod \"service-ca-operator-57b5b68b8-jc96l\" (UID: \"d5a29851-cc12-4af0-93df-a15c86f0ee86\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.747972 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.747947 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2lm\" (UniqueName: \"kubernetes.io/projected/077160a2-2714-4cf9-961e-4169dc361eaa-kube-api-access-xz2lm\") pod \"console-operator-76dff48d59-jfbcj\" (UID: \"077160a2-2714-4cf9-961e-4169dc361eaa\") " pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.748059 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.748039 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzgk\" (UniqueName: \"kubernetes.io/projected/6aeb4909-83b4-4b8a-ac96-d20a74970d44-kube-api-access-rqzgk\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:02.748383 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.748369 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksst\" (UniqueName: \"kubernetes.io/projected/97a552c2-8f68-4558-85d3-300397fe201b-kube-api-access-mksst\") pod \"volume-data-source-validator-5cf8c89d64-8ngrf\" (UID: \"97a552c2-8f68-4558-85d3-300397fe201b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" May 25 13:08:02.832901 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.832871 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/720a0198-112a-4272-b799-e51dbaf56832-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.833036 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.832962 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/720a0198-112a-4272-b799-e51dbaf56832-config\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.833036 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.832990 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6gs\" (UniqueName: \"kubernetes.io/projected/720a0198-112a-4272-b799-e51dbaf56832-kube-api-access-qx6gs\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.833474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.833454 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/720a0198-112a-4272-b799-e51dbaf56832-config\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.834921 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.834903 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/720a0198-112a-4272-b799-e51dbaf56832-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.843675 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.843653 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6gs\" (UniqueName: \"kubernetes.io/projected/720a0198-112a-4272-b799-e51dbaf56832-kube-api-access-qx6gs\") pod \"kube-storage-version-migrator-operator-565644f49-9669r\" (UID: \"720a0198-112a-4272-b799-e51dbaf56832\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.846464 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.846446 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" May 25 13:08:02.864475 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.864448 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:02.870629 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.870608 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-vt6mr"] May 25 13:08:02.873795 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:02.873770 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d027ad_c810_423a_98fa_54547bad1795.slice/crio-1eeafe074d0004db5cab25230253fe2bb8fb44df2ca7d7e4d8608eed2de41413 WatchSource:0}: Error finding container 1eeafe074d0004db5cab25230253fe2bb8fb44df2ca7d7e4d8608eed2de41413: Status 404 returned error can't find the container with id 1eeafe074d0004db5cab25230253fe2bb8fb44df2ca7d7e4d8608eed2de41413 May 25 13:08:02.879149 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.879129 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" May 25 13:08:02.975081 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.975019 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" May 25 13:08:02.984900 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:02.984732 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l"] May 25 13:08:02.987888 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:02.987863 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a29851_cc12_4af0_93df_a15c86f0ee86.slice/crio-d2130caee33f207234fd5d8184933913b12889355f24766f0524086d414ec258 WatchSource:0}: Error finding container d2130caee33f207234fd5d8184933913b12889355f24766f0524086d414ec258: Status 404 returned error can't find the container with id d2130caee33f207234fd5d8184933913b12889355f24766f0524086d414ec258 May 25 13:08:03.003406 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.003375 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-jfbcj"] May 25 13:08:03.007790 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:03.007759 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077160a2_2714_4cf9_961e_4169dc361eaa.slice/crio-ca6ddbe68c287f29baf3396abffb65cf5bc306b005ff42027312eac357c13521 WatchSource:0}: Error finding container ca6ddbe68c287f29baf3396abffb65cf5bc306b005ff42027312eac357c13521: Status 404 returned error can't find the container with id ca6ddbe68c287f29baf3396abffb65cf5bc306b005ff42027312eac357c13521 May 25 13:08:03.027182 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.027064 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf"] May 25 13:08:03.030990 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:03.030947 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a552c2_8f68_4558_85d3_300397fe201b.slice/crio-4c721b447cc6cdf6eff056be24f73b00676e6804f15cccd711c1bd7ae65ce5c7 WatchSource:0}: Error finding container 4c721b447cc6cdf6eff056be24f73b00676e6804f15cccd711c1bd7ae65ce5c7: Status 404 returned error can't find the container with id 4c721b447cc6cdf6eff056be24f73b00676e6804f15cccd711c1bd7ae65ce5c7 May 25 13:08:03.097331 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.097305 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r"] May 25 13:08:03.100159 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:03.100130 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720a0198_112a_4272_b799_e51dbaf56832.slice/crio-c79de597b2cfd47bb80b1d10e1abc8e095a40d8b702d7f6722e86f1614e8273f WatchSource:0}: Error finding container c79de597b2cfd47bb80b1d10e1abc8e095a40d8b702d7f6722e86f1614e8273f: Status 404 returned error can't find the container with id c79de597b2cfd47bb80b1d10e1abc8e095a40d8b702d7f6722e86f1614e8273f May 25 13:08:03.133954 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.133930 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" event={"ID":"97a552c2-8f68-4558-85d3-300397fe201b","Type":"ContainerStarted","Data":"4c721b447cc6cdf6eff056be24f73b00676e6804f15cccd711c1bd7ae65ce5c7"} May 25 13:08:03.134765 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.134744 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" event={"ID":"720a0198-112a-4272-b799-e51dbaf56832","Type":"ContainerStarted","Data":"c79de597b2cfd47bb80b1d10e1abc8e095a40d8b702d7f6722e86f1614e8273f"} May 25 13:08:03.135706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.135688 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" event={"ID":"077160a2-2714-4cf9-961e-4169dc361eaa","Type":"ContainerStarted","Data":"ca6ddbe68c287f29baf3396abffb65cf5bc306b005ff42027312eac357c13521"} May 25 13:08:03.136534 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.136515 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" event={"ID":"d5a29851-cc12-4af0-93df-a15c86f0ee86","Type":"ContainerStarted","Data":"d2130caee33f207234fd5d8184933913b12889355f24766f0524086d414ec258"} May 25 13:08:03.137374 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.137356 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" event={"ID":"69d027ad-c810-423a-98fa-54547bad1795","Type":"ContainerStarted","Data":"1eeafe074d0004db5cab25230253fe2bb8fb44df2ca7d7e4d8608eed2de41413"} May 25 13:08:03.236244 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.236173 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:03.236362 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:03.236244 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:03.236362 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:03.236317 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 25 13:08:03.236462 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:03.236377 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls podName:6aeb4909-83b4-4b8a-ac96-d20a74970d44 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:04.236362361 +0000 UTC m=+95.914659941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-zb92d" (UID: "6aeb4909-83b4-4b8a-ac96-d20a74970d44") : secret "cluster-monitoring-operator-tls" not found May 25 13:08:03.236462 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:03.236383 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 25 13:08:03.236462 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:03.236449 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls podName:3df04ad1-0e90-459d-b66b-a391ef478d0f nodeName:}" failed. No retries permitted until 2026-05-25 13:08:04.236417165 +0000 UTC m=+95.914714744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls") pod "cluster-samples-operator-697587599-v7h8q" (UID: "3df04ad1-0e90-459d-b66b-a391ef478d0f") : secret "samples-operator-tls" not found May 25 13:08:04.039751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.038719 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-646f7c65f5-v9krd"] May 25 13:08:04.043611 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.040916 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.047384 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.044728 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 25 13:08:04.050234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.049663 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 25 13:08:04.050234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.049897 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 25 13:08:04.050234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.050074 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mf8zb\"" May 25 13:08:04.050714 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.050691 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 25 13:08:04.060528 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.060502 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646f7c65f5-v9krd"] May 25 13:08:04.146119 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146089 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7fa88-8808-4d44-b234-377a131ed4b3-ca-trust-extracted\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.146266 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146132 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-image-registry-private-configuration\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.146266 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-certificates\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.146266 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-bound-sa-token\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.146452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146269 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-installation-pull-secrets\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.146452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146323 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.146452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146347 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-trusted-ca\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.146452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.146384 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhfk\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-kube-api-access-6vhfk\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.247630 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247593 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7fa88-8808-4d44-b234-377a131ed4b3-ca-trust-extracted\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.247782 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247644 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-image-registry-private-configuration\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.247782 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247680 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-certificates\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.247782 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247710 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-bound-sa-token\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.247782 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247749 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-installation-pull-secrets\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.248062 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247787 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.248062 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247813 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-trusted-ca\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.248062 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247849 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhfk\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-kube-api-access-6vhfk\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.248062 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247884 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:04.248062 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.247926 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:04.248460 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.248066 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 25 13:08:04.248460 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.248125 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls podName:3df04ad1-0e90-459d-b66b-a391ef478d0f nodeName:}" failed. No retries permitted until 2026-05-25 13:08:06.248107077 +0000 UTC m=+97.926404672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls") pod "cluster-samples-operator-697587599-v7h8q" (UID: "3df04ad1-0e90-459d-b66b-a391ef478d0f") : secret "samples-operator-tls" not found May 25 13:08:04.248847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.248823 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7fa88-8808-4d44-b234-377a131ed4b3-ca-trust-extracted\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.249349 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.249329 2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 25 13:08:04.249349 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.249350 2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646f7c65f5-v9krd: secret "image-registry-tls" not found May 25 13:08:04.249492 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.249397 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls podName:d8d7fa88-8808-4d44-b234-377a131ed4b3 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:04.74938044 +0000 UTC m=+96.427678024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls") pod "image-registry-646f7c65f5-v9krd" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3") : secret "image-registry-tls" not found May 25 13:08:04.249978 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.249955 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-certificates\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.252514 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.251658 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 25 13:08:04.252514 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.251720 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls podName:6aeb4909-83b4-4b8a-ac96-d20a74970d44 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:06.251704102 +0000 UTC m=+97.930001688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-zb92d" (UID: "6aeb4909-83b4-4b8a-ac96-d20a74970d44") : secret "cluster-monitoring-operator-tls" not found May 25 13:08:04.252514 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.252387 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-trusted-ca\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.253353 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.253311 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-image-registry-private-configuration\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.256619 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.256573 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-installation-pull-secrets\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.270476 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.270352 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhfk\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-kube-api-access-6vhfk\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.281636 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.281596 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-bound-sa-token\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.751823 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:04.751779 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:04.751983 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.751954 2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 25 13:08:04.751983 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.751978 2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646f7c65f5-v9krd: secret "image-registry-tls" not found May 25 13:08:04.752110 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:04.752050 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls podName:d8d7fa88-8808-4d44-b234-377a131ed4b3 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:05.752028987 +0000 UTC m=+97.430326573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls") pod "image-registry-646f7c65f5-v9krd" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3") : secret "image-registry-tls" not found May 25 13:08:05.761677 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:05.761637 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:05.762096 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:05.761779 2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 25 13:08:05.762096 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:05.761803 2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646f7c65f5-v9krd: secret "image-registry-tls" not found May 25 13:08:05.762096 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:05.761871 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls podName:d8d7fa88-8808-4d44-b234-377a131ed4b3 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:07.761850551 +0000 UTC m=+99.440148136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls") pod "image-registry-646f7c65f5-v9krd" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3") : secret "image-registry-tls" not found May 25 13:08:06.064402 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:06.064318 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:08:06.064574 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:06.064451 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:08:06.064574 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.064456 2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 25 13:08:06.064574 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.064520 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert podName:232bc31b-feac-4e2b-9fb3-f1d3c3a0a075 nodeName:}" failed. No retries permitted until 2026-05-25 13:09:10.06449798 +0000 UTC m=+161.742795560 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert") pod "ingress-canary-64d4d" (UID: "232bc31b-feac-4e2b-9fb3-f1d3c3a0a075") : secret "canary-serving-cert" not found May 25 13:08:06.064574 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.064539 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 25 13:08:06.064574 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.064576 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls podName:a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8 nodeName:}" failed. No retries permitted until 2026-05-25 13:09:10.064565547 +0000 UTC m=+161.742863127 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls") pod "dns-default-h8xqz" (UID: "a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8") : secret "dns-default-metrics-tls" not found May 25 13:08:06.266529 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:06.266494 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:06.266679 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:06.266550 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:06.266679 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.266647 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 25 13:08:06.266760 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.266722 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 25 13:08:06.266807 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.266725 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls podName:6aeb4909-83b4-4b8a-ac96-d20a74970d44 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:10.266708256 +0000 UTC m=+101.945005836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-zb92d" (UID: "6aeb4909-83b4-4b8a-ac96-d20a74970d44") : secret "cluster-monitoring-operator-tls" not found May 25 13:08:06.266807 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:06.266787 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls podName:3df04ad1-0e90-459d-b66b-a391ef478d0f nodeName:}" failed. No retries permitted until 2026-05-25 13:08:10.266768098 +0000 UTC m=+101.945065683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls") pod "cluster-samples-operator-697587599-v7h8q" (UID: "3df04ad1-0e90-459d-b66b-a391ef478d0f") : secret "samples-operator-tls" not found May 25 13:08:07.149232 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.149200 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" event={"ID":"720a0198-112a-4272-b799-e51dbaf56832","Type":"ContainerStarted","Data":"dce081fcf5af45759ddbf20f5d6ee914ff2757f9779575dbe5a3b51b5158fb3f"} May 25 13:08:07.150745 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.150723 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/0.log" May 25 13:08:07.150836 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.150767 2554 generic.go:358] "Generic (PLEG): container finished" podID="077160a2-2714-4cf9-961e-4169dc361eaa" containerID="b8b96be8f5fd2f9726dcc07823fc1ee354e2940193529fc278abf70f7fc8ee67" exitCode=255 May 25 13:08:07.150881 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.150836 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" event={"ID":"077160a2-2714-4cf9-961e-4169dc361eaa","Type":"ContainerDied","Data":"b8b96be8f5fd2f9726dcc07823fc1ee354e2940193529fc278abf70f7fc8ee67"} May 25 13:08:07.151042 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.151018 2554 scope.go:117] "RemoveContainer" containerID="b8b96be8f5fd2f9726dcc07823fc1ee354e2940193529fc278abf70f7fc8ee67" May 25 13:08:07.152102 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.152077 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" event={"ID":"d5a29851-cc12-4af0-93df-a15c86f0ee86","Type":"ContainerStarted","Data":"4db3c753e577af0e9ded0d01dcd64fba61ae6cb9079e29dc1216b16ec714f4dd"} May 25 13:08:07.153529 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.153501 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" event={"ID":"69d027ad-c810-423a-98fa-54547bad1795","Type":"ContainerStarted","Data":"ddff919abaa53c7fc96063c68e6d5ce82b11b2dcb4bd3d3a99476312b8f394e0"} May 25 13:08:07.154790 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.154770 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" event={"ID":"97a552c2-8f68-4558-85d3-300397fe201b","Type":"ContainerStarted","Data":"b3828631898c47af62cfde8a94e14c93aeeaa94600e7cacaea46a440a78e64c2"} May 25 13:08:07.169807 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.169750 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" podStartSLOduration=1.556340219 podStartE2EDuration="5.169735744s" podCreationTimestamp="2026-05-25 13:08:02 +0000 UTC" firstStartedPulling="2026-05-25 13:08:03.101916714 +0000 UTC m=+94.780214297" lastFinishedPulling="2026-05-25 13:08:06.715312229 +0000 UTC m=+98.393609822" observedRunningTime="2026-05-25 13:08:07.169150207 +0000 UTC m=+98.847447810" watchObservedRunningTime="2026-05-25 13:08:07.169735744 +0000 UTC m=+98.848033349" May 25 13:08:07.202852 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.202803 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-8ngrf" podStartSLOduration=1.523230023 podStartE2EDuration="5.202766975s" podCreationTimestamp="2026-05-25 13:08:02 +0000 UTC" firstStartedPulling="2026-05-25 13:08:03.033076185 +0000 UTC m=+94.711373764" lastFinishedPulling="2026-05-25 13:08:06.712613124 +0000 UTC m=+98.390910716" observedRunningTime="2026-05-25 13:08:07.202142676 +0000 UTC m=+98.880440279" watchObservedRunningTime="2026-05-25 13:08:07.202766975 +0000 UTC m=+98.881064579" May 25 13:08:07.224383 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.224339 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" podStartSLOduration=1.383735993 podStartE2EDuration="5.224323395s" podCreationTimestamp="2026-05-25 13:08:02 +0000 UTC" firstStartedPulling="2026-05-25 13:08:02.875750112 +0000 UTC m=+94.554047692" lastFinishedPulling="2026-05-25 13:08:06.716337511 +0000 UTC m=+98.394635094" observedRunningTime="2026-05-25 13:08:07.224281127 +0000 UTC m=+98.902578753" watchObservedRunningTime="2026-05-25 13:08:07.224323395 +0000 UTC m=+98.902620996" May 25 13:08:07.243606 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.243561 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" podStartSLOduration=1.51190132 podStartE2EDuration="5.243544393s" podCreationTimestamp="2026-05-25 13:08:02 +0000 UTC" firstStartedPulling="2026-05-25 13:08:02.990177238 +0000 UTC m=+94.668474832" lastFinishedPulling="2026-05-25 13:08:06.721820314 +0000 UTC m=+98.400117905" observedRunningTime="2026-05-25 13:08:07.242277139 +0000 UTC m=+98.920574742" watchObservedRunningTime="2026-05-25 13:08:07.243544393 +0000 UTC m=+98.921842000" May 25 13:08:07.781116 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:07.781081 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:07.781359 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:07.781198 2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 25 13:08:07.781359 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:07.781211 2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646f7c65f5-v9krd: secret "image-registry-tls" not found May 25 13:08:07.781359 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:07.781260 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls podName:d8d7fa88-8808-4d44-b234-377a131ed4b3 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:11.781246695 +0000 UTC m=+103.459544274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls") pod "image-registry-646f7c65f5-v9krd" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3") : secret "image-registry-tls" not found May 25 13:08:08.038906 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.038832 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4"] May 25 13:08:08.041214 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.041197 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" May 25 13:08:08.043504 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.043484 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f5dgw\"" May 25 13:08:08.044254 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.044240 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" May 25 13:08:08.044322 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.044289 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" May 25 13:08:08.052958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.052937 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4"] May 25 13:08:08.158196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.158174 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/1.log" May 25 13:08:08.158565 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.158549 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/0.log" May 25 13:08:08.158614 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.158585 2554 generic.go:358] "Generic (PLEG): container finished" podID="077160a2-2714-4cf9-961e-4169dc361eaa" containerID="ddbc9b56aa65762e83e798caf0013951c88728b165333991f9d3ec34c38427f0" exitCode=255 May 25 13:08:08.158735 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.158707 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" event={"ID":"077160a2-2714-4cf9-961e-4169dc361eaa","Type":"ContainerDied","Data":"ddbc9b56aa65762e83e798caf0013951c88728b165333991f9d3ec34c38427f0"} May 25 13:08:08.158806 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.158754 2554 scope.go:117] "RemoveContainer" containerID="b8b96be8f5fd2f9726dcc07823fc1ee354e2940193529fc278abf70f7fc8ee67" May 25 13:08:08.158915 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.158901 2554 scope.go:117] "RemoveContainer" containerID="ddbc9b56aa65762e83e798caf0013951c88728b165333991f9d3ec34c38427f0" May 25 13:08:08.159143 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:08.159105 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-jfbcj_openshift-console-operator(077160a2-2714-4cf9-961e-4169dc361eaa)\"" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" podUID="077160a2-2714-4cf9-961e-4169dc361eaa" May 25 13:08:08.185116 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.185096 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79k2\" (UniqueName: \"kubernetes.io/projected/e1670f1d-70e9-440c-93f4-daf299a206d8-kube-api-access-l79k2\") pod \"migrator-b985f5d89-5sqj4\" (UID: \"e1670f1d-70e9-440c-93f4-daf299a206d8\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" May 25 13:08:08.286276 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.286251 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l79k2\" (UniqueName: \"kubernetes.io/projected/e1670f1d-70e9-440c-93f4-daf299a206d8-kube-api-access-l79k2\") pod \"migrator-b985f5d89-5sqj4\" (UID: \"e1670f1d-70e9-440c-93f4-daf299a206d8\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" May 25 13:08:08.293800 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.293745 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79k2\" (UniqueName: \"kubernetes.io/projected/e1670f1d-70e9-440c-93f4-daf299a206d8-kube-api-access-l79k2\") pod \"migrator-b985f5d89-5sqj4\" (UID: \"e1670f1d-70e9-440c-93f4-daf299a206d8\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" May 25 13:08:08.349859 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.349839 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" May 25 13:08:08.464577 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:08.464552 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4"] May 25 13:08:08.467404 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:08.467377 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1670f1d_70e9_440c_93f4_daf299a206d8.slice/crio-1b4fd6d513a1e54911203ab06341f8bd5d3a8068e5ac9fa39d231a8702785296 WatchSource:0}: Error finding container 1b4fd6d513a1e54911203ab06341f8bd5d3a8068e5ac9fa39d231a8702785296: Status 404 returned error can't find the container with id 1b4fd6d513a1e54911203ab06341f8bd5d3a8068e5ac9fa39d231a8702785296 May 25 13:08:09.091882 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:09.091853 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bhlgr" May 25 13:08:09.163198 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:09.163163 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" event={"ID":"e1670f1d-70e9-440c-93f4-daf299a206d8","Type":"ContainerStarted","Data":"1b4fd6d513a1e54911203ab06341f8bd5d3a8068e5ac9fa39d231a8702785296"} May 25 13:08:09.164975 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:09.164954 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/1.log" May 25 13:08:09.165332 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:09.165299 2554 scope.go:117] "RemoveContainer" containerID="ddbc9b56aa65762e83e798caf0013951c88728b165333991f9d3ec34c38427f0" May 25 13:08:09.165610 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:09.165580 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-jfbcj_openshift-console-operator(077160a2-2714-4cf9-961e-4169dc361eaa)\"" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" podUID="077160a2-2714-4cf9-961e-4169dc361eaa" May 25 13:08:09.570440 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:09.570395 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d5p4k_e460f6bd-9a46-46d2-870f-f4d203cf05fd/dns-node-resolver/0.log" May 25 13:08:10.168976 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:10.168943 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" event={"ID":"e1670f1d-70e9-440c-93f4-daf299a206d8","Type":"ContainerStarted","Data":"6d46e06b5ec6950fa8ccba6afb4b54c515e70bf05a78cbd07c4365763145c0d7"} May 25 13:08:10.169314 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:10.168982 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" event={"ID":"e1670f1d-70e9-440c-93f4-daf299a206d8","Type":"ContainerStarted","Data":"55732d089a4700673fc563bcba31787644260371ed90e10beeb819d60e4c9f88"} May 25 13:08:10.186592 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:10.186552 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-5sqj4" podStartSLOduration=1.036095865 podStartE2EDuration="2.18653784s" podCreationTimestamp="2026-05-25 13:08:08 +0000 UTC" firstStartedPulling="2026-05-25 13:08:08.469297878 +0000 UTC m=+100.147595462" lastFinishedPulling="2026-05-25 13:08:09.619739843 +0000 UTC m=+101.298037437" observedRunningTime="2026-05-25 13:08:10.185198577 +0000 UTC m=+101.863496181" watchObservedRunningTime="2026-05-25 13:08:10.18653784 +0000 UTC m=+101.864835505" May 25 13:08:10.303566 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:10.303539 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:10.303684 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:10.303672 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:10.303727 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:10.303686 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 25 13:08:10.303761 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:10.303751 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls podName:3df04ad1-0e90-459d-b66b-a391ef478d0f nodeName:}" failed. No retries permitted until 2026-05-25 13:08:18.303732559 +0000 UTC m=+109.982030156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls") pod "cluster-samples-operator-697587599-v7h8q" (UID: "3df04ad1-0e90-459d-b66b-a391ef478d0f") : secret "samples-operator-tls" not found May 25 13:08:10.303761 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:10.303754 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 25 13:08:10.303832 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:10.303790 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls podName:6aeb4909-83b4-4b8a-ac96-d20a74970d44 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:18.303780251 +0000 UTC m=+109.982077834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-zb92d" (UID: "6aeb4909-83b4-4b8a-ac96-d20a74970d44") : secret "cluster-monitoring-operator-tls" not found May 25 13:08:10.571166 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:10.571142 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5brhb_db1bb1ec-d3b1-43f1-b351-de93dee1b1ea/node-ca/0.log" May 25 13:08:11.814603 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:11.814572 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:11.814989 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:11.814715 2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 25 13:08:11.814989 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:11.814737 2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-646f7c65f5-v9krd: secret "image-registry-tls" not found May 25 13:08:11.814989 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:11.814789 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls podName:d8d7fa88-8808-4d44-b234-377a131ed4b3 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:19.814775015 +0000 UTC m=+111.493072595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls") pod "image-registry-646f7c65f5-v9krd" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3") : secret "image-registry-tls" not found May 25 13:08:12.865519 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:12.865490 2554 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:12.865519 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:12.865526 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:12.865924 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:12.865863 2554 scope.go:117] "RemoveContainer" containerID="ddbc9b56aa65762e83e798caf0013951c88728b165333991f9d3ec34c38427f0" May 25 13:08:12.866049 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:12.866018 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-jfbcj_openshift-console-operator(077160a2-2714-4cf9-961e-4169dc361eaa)\"" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" podUID="077160a2-2714-4cf9-961e-4169dc361eaa" May 25 13:08:18.360996 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:18.360952 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:18.360996 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:18.361002 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:18.361551 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:18.361100 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 25 13:08:18.361551 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:18.361175 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls podName:6aeb4909-83b4-4b8a-ac96-d20a74970d44 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:34.361156116 +0000 UTC m=+126.039453697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-zb92d" (UID: "6aeb4909-83b4-4b8a-ac96-d20a74970d44") : secret "cluster-monitoring-operator-tls" not found May 25 13:08:18.363702 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:18.363676 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3df04ad1-0e90-459d-b66b-a391ef478d0f-samples-operator-tls\") pod \"cluster-samples-operator-697587599-v7h8q\" (UID: \"3df04ad1-0e90-459d-b66b-a391ef478d0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:18.459915 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:18.459876 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" May 25 13:08:18.581972 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:18.581947 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q"] May 25 13:08:19.193144 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:19.193094 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" event={"ID":"3df04ad1-0e90-459d-b66b-a391ef478d0f","Type":"ContainerStarted","Data":"7aafd07c2f1392819a653f403054d5ba7c9f20869614f1f70b786cce0d53cd1f"} May 25 13:08:19.872884 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:19.872848 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:19.875299 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:19.875276 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"image-registry-646f7c65f5-v9krd\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:19.957129 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:19.957098 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:20.164659 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:20.164636 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646f7c65f5-v9krd"] May 25 13:08:20.169734 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:20.169706 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d7fa88_8808_4d44_b234_377a131ed4b3.slice/crio-abea31cf257fd519d0202fcb6104df4c1a191d068191f45893d25224b16faada WatchSource:0}: Error finding container abea31cf257fd519d0202fcb6104df4c1a191d068191f45893d25224b16faada: Status 404 returned error can't find the container with id abea31cf257fd519d0202fcb6104df4c1a191d068191f45893d25224b16faada May 25 13:08:20.198325 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:20.198245 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" event={"ID":"3df04ad1-0e90-459d-b66b-a391ef478d0f","Type":"ContainerStarted","Data":"30b80f56f61450b071fa6a97d89a226baccbbef4d46a3350559e9fa1e4677c46"} May 25 13:08:20.199582 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:20.199527 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" event={"ID":"d8d7fa88-8808-4d44-b234-377a131ed4b3","Type":"ContainerStarted","Data":"abea31cf257fd519d0202fcb6104df4c1a191d068191f45893d25224b16faada"} May 25 13:08:21.203440 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:21.203392 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" event={"ID":"d8d7fa88-8808-4d44-b234-377a131ed4b3","Type":"ContainerStarted","Data":"c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222"} May 25 13:08:21.203834 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:21.203511 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:21.204981 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:21.204957 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" event={"ID":"3df04ad1-0e90-459d-b66b-a391ef478d0f","Type":"ContainerStarted","Data":"cab2937ee30cc00957443af8e6225f4577a9083a476621d2b1951103adbf15ed"} May 25 13:08:21.222896 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:21.222855 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" podStartSLOduration=17.222842751 podStartE2EDuration="17.222842751s" podCreationTimestamp="2026-05-25 13:08:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:08:21.221464684 +0000 UTC m=+112.899762285" watchObservedRunningTime="2026-05-25 13:08:21.222842751 +0000 UTC m=+112.901140353" May 25 13:08:21.239642 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:21.239601 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-v7h8q" podStartSLOduration=17.792266038 podStartE2EDuration="19.239589547s" podCreationTimestamp="2026-05-25 13:08:02 +0000 UTC" firstStartedPulling="2026-05-25 13:08:18.621632052 +0000 UTC m=+110.299929635" lastFinishedPulling="2026-05-25 13:08:20.06895556 +0000 UTC m=+111.747253144" observedRunningTime="2026-05-25 13:08:21.238793713 +0000 UTC m=+112.917091316" watchObservedRunningTime="2026-05-25 13:08:21.239589547 +0000 UTC m=+112.917887213" May 25 13:08:27.896881 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:27.896849 2554 scope.go:117] "RemoveContainer" containerID="ddbc9b56aa65762e83e798caf0013951c88728b165333991f9d3ec34c38427f0" May 25 13:08:28.225463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:28.225393 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:08:28.225764 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:28.225745 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/1.log" May 25 13:08:28.225854 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:28.225784 2554 generic.go:358] "Generic (PLEG): container finished" podID="077160a2-2714-4cf9-961e-4169dc361eaa" containerID="7293e6649187cc2060c49908eb998920aa249c4ef3bfe70c86a61ee88001314e" exitCode=255 May 25 13:08:28.225854 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:28.225832 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" event={"ID":"077160a2-2714-4cf9-961e-4169dc361eaa","Type":"ContainerDied","Data":"7293e6649187cc2060c49908eb998920aa249c4ef3bfe70c86a61ee88001314e"} May 25 13:08:28.225958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:28.225872 2554 scope.go:117] "RemoveContainer" containerID="ddbc9b56aa65762e83e798caf0013951c88728b165333991f9d3ec34c38427f0" May 25 13:08:28.226202 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:28.226186 2554 scope.go:117] "RemoveContainer" containerID="7293e6649187cc2060c49908eb998920aa249c4ef3bfe70c86a61ee88001314e" May 25 13:08:28.226412 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:28.226394 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-jfbcj_openshift-console-operator(077160a2-2714-4cf9-961e-4169dc361eaa)\"" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" podUID="077160a2-2714-4cf9-961e-4169dc361eaa" May 25 13:08:29.229655 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:29.229627 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:08:30.594245 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.594217 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tg88x"] May 25 13:08:30.598916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.598892 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.606397 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.606368 2554 status_manager.go:895] "Failed to get status for pod" podUID="17da813c-56ef-431e-906e-02d26698b3ab" pod="openshift-insights/insights-runtime-extractor-tg88x" err="pods \"insights-runtime-extractor-tg88x\" is forbidden: User \"system:node:ip-10-0-135-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-64.ec2.internal' and this object" May 25 13:08:30.606849 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:30.606823 2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-135-64.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-64.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" May 25 13:08:30.611790 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:30.611757 2554 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-135-64.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-64.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" May 25 13:08:30.612005 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:30.611979 2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-84t4d\" is forbidden: User \"system:node:ip-10-0-135-64.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-64.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-84t4d\"" type="*v1.Secret" May 25 13:08:30.633899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.633873 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tg88x"] May 25 13:08:30.656797 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.656777 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-646f7c65f5-v9krd"] May 25 13:08:30.713595 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.713570 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78f78dfc48-kpjv4"] May 25 13:08:30.716640 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.716624 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.732636 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.732615 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f78dfc48-kpjv4"] May 25 13:08:30.752115 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.752095 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17da813c-56ef-431e-906e-02d26698b3ab-crio-socket\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.752208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.752121 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17da813c-56ef-431e-906e-02d26698b3ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.752208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.752156 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17da813c-56ef-431e-906e-02d26698b3ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.752295 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.752258 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdv9n\" (UniqueName: \"kubernetes.io/projected/17da813c-56ef-431e-906e-02d26698b3ab-kube-api-access-hdv9n\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.752343 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.752299 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17da813c-56ef-431e-906e-02d26698b3ab-data-volume\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.853394 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853344 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hdv9n\" (UniqueName: \"kubernetes.io/projected/17da813c-56ef-431e-906e-02d26698b3ab-kube-api-access-hdv9n\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.853394 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853379 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17da813c-56ef-431e-906e-02d26698b3ab-data-volume\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.853540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853402 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df3bdbc0-4ecb-4a46-9efa-717313373f35-trusted-ca\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853420 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df3bdbc0-4ecb-4a46-9efa-717313373f35-registry-certificates\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853477 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df3bdbc0-4ecb-4a46-9efa-717313373f35-ca-trust-extracted\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853532 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtgd\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-kube-api-access-vvtgd\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853694 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853550 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-registry-tls\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853694 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853579 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-bound-sa-token\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853694 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853645 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17da813c-56ef-431e-906e-02d26698b3ab-crio-socket\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.853694 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853673 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17da813c-56ef-431e-906e-02d26698b3ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.853845 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853707 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df3bdbc0-4ecb-4a46-9efa-717313373f35-installation-pull-secrets\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853845 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853721 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17da813c-56ef-431e-906e-02d26698b3ab-crio-socket\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.853845 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853751 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17da813c-56ef-431e-906e-02d26698b3ab-data-volume\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.853845 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853753 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df3bdbc0-4ecb-4a46-9efa-717313373f35-image-registry-private-configuration\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.853845 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.853789 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17da813c-56ef-431e-906e-02d26698b3ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.861926 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.861904 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdv9n\" (UniqueName: \"kubernetes.io/projected/17da813c-56ef-431e-906e-02d26698b3ab-kube-api-access-hdv9n\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:30.954258 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954232 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df3bdbc0-4ecb-4a46-9efa-717313373f35-trusted-ca\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954364 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954262 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df3bdbc0-4ecb-4a46-9efa-717313373f35-registry-certificates\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954364 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954280 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df3bdbc0-4ecb-4a46-9efa-717313373f35-ca-trust-extracted\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954364 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954297 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtgd\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-kube-api-access-vvtgd\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954364 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954314 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-registry-tls\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954364 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954342 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-bound-sa-token\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954659 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954375 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df3bdbc0-4ecb-4a46-9efa-717313373f35-installation-pull-secrets\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954659 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954411 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df3bdbc0-4ecb-4a46-9efa-717313373f35-image-registry-private-configuration\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.954759 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.954722 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df3bdbc0-4ecb-4a46-9efa-717313373f35-ca-trust-extracted\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.955277 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.955257 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df3bdbc0-4ecb-4a46-9efa-717313373f35-registry-certificates\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.955372 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.955331 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df3bdbc0-4ecb-4a46-9efa-717313373f35-trusted-ca\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.956857 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.956834 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df3bdbc0-4ecb-4a46-9efa-717313373f35-installation-pull-secrets\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.957195 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.957175 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df3bdbc0-4ecb-4a46-9efa-717313373f35-image-registry-private-configuration\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.957314 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.957295 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-registry-tls\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.963635 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.963611 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtgd\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-kube-api-access-vvtgd\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:30.965082 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:30.965061 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df3bdbc0-4ecb-4a46-9efa-717313373f35-bound-sa-token\") pod \"image-registry-78f78dfc48-kpjv4\" (UID: \"df3bdbc0-4ecb-4a46-9efa-717313373f35\") " pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:31.024600 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.024575 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:31.151796 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.151722 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f78dfc48-kpjv4"] May 25 13:08:31.154243 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:31.154213 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3bdbc0_4ecb_4a46_9efa_717313373f35.slice/crio-819a051fdd8dff1d169dcf7e11ddfbc550ccb8b7821cc69f92d338984cb67805 WatchSource:0}: Error finding container 819a051fdd8dff1d169dcf7e11ddfbc550ccb8b7821cc69f92d338984cb67805: Status 404 returned error can't find the container with id 819a051fdd8dff1d169dcf7e11ddfbc550ccb8b7821cc69f92d338984cb67805 May 25 13:08:31.236005 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.235977 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" event={"ID":"df3bdbc0-4ecb-4a46-9efa-717313373f35","Type":"ContainerStarted","Data":"326af2027c9d0c854822639e35a4ce3f6b97eda7149d7b205f4af0712b5cc107"} May 25 13:08:31.236005 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.236010 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" event={"ID":"df3bdbc0-4ecb-4a46-9efa-717313373f35","Type":"ContainerStarted","Data":"819a051fdd8dff1d169dcf7e11ddfbc550ccb8b7821cc69f92d338984cb67805"} May 25 13:08:31.236180 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.236094 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:31.254967 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.254929 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" podStartSLOduration=1.254916726 podStartE2EDuration="1.254916726s" podCreationTimestamp="2026-05-25 13:08:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:08:31.253223897 +0000 UTC m=+122.931521499" watchObservedRunningTime="2026-05-25 13:08:31.254916726 +0000 UTC m=+122.933214321" May 25 13:08:31.582868 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.582840 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 25 13:08:31.586816 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.586794 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17da813c-56ef-431e-906e-02d26698b3ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:31.854244 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:31.854181 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: failed to sync configmap cache: timed out waiting for the condition May 25 13:08:31.854587 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:31.854261 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17da813c-56ef-431e-906e-02d26698b3ab-kube-rbac-proxy-cm podName:17da813c-56ef-431e-906e-02d26698b3ab nodeName:}" failed. No retries permitted until 2026-05-25 13:08:32.354245587 +0000 UTC m=+124.032543167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17da813c-56ef-431e-906e-02d26698b3ab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tg88x" (UID: "17da813c-56ef-431e-906e-02d26698b3ab") : failed to sync configmap cache: timed out waiting for the condition May 25 13:08:31.911740 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.911716 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 25 13:08:31.952339 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:31.952314 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-84t4d\"" May 25 13:08:32.364504 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:32.364474 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17da813c-56ef-431e-906e-02d26698b3ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:32.365039 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:32.365016 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17da813c-56ef-431e-906e-02d26698b3ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg88x\" (UID: \"17da813c-56ef-431e-906e-02d26698b3ab\") " pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:32.406997 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:32.406972 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tg88x" May 25 13:08:32.528909 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:32.528883 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tg88x"] May 25 13:08:32.531605 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:32.531577 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17da813c_56ef_431e_906e_02d26698b3ab.slice/crio-3bea26673ad077aa4890222ea84cc1870db87d0100838c58a1b1b1b5c57c735b WatchSource:0}: Error finding container 3bea26673ad077aa4890222ea84cc1870db87d0100838c58a1b1b1b5c57c735b: Status 404 returned error can't find the container with id 3bea26673ad077aa4890222ea84cc1870db87d0100838c58a1b1b1b5c57c735b May 25 13:08:32.865033 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:32.865007 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:32.865372 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:32.865045 2554 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:32.865372 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:32.865342 2554 scope.go:117] "RemoveContainer" containerID="7293e6649187cc2060c49908eb998920aa249c4ef3bfe70c86a61ee88001314e" May 25 13:08:32.865534 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:32.865515 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-jfbcj_openshift-console-operator(077160a2-2714-4cf9-961e-4169dc361eaa)\"" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" podUID="077160a2-2714-4cf9-961e-4169dc361eaa" May 25 13:08:33.243220 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:33.243187 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg88x" event={"ID":"17da813c-56ef-431e-906e-02d26698b3ab","Type":"ContainerStarted","Data":"7cd7f31969b440b800443655006b9acc9353d91cc83ec196bc1a8f54496a0f1e"} May 25 13:08:33.243220 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:33.243222 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg88x" event={"ID":"17da813c-56ef-431e-906e-02d26698b3ab","Type":"ContainerStarted","Data":"5a17e997be0ba8414e4c2522971546f293948ced499fb4df91d232d89753ed4b"} May 25 13:08:33.243380 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:33.243231 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg88x" event={"ID":"17da813c-56ef-431e-906e-02d26698b3ab","Type":"ContainerStarted","Data":"3bea26673ad077aa4890222ea84cc1870db87d0100838c58a1b1b1b5c57c735b"} May 25 13:08:34.378998 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:34.378976 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:34.381077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:34.381048 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6aeb4909-83b4-4b8a-ac96-d20a74970d44-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-zb92d\" (UID: \"6aeb4909-83b4-4b8a-ac96-d20a74970d44\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:34.656104 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:34.656030 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wmwhg\"" May 25 13:08:34.663995 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:34.663980 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" May 25 13:08:34.780725 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:34.780694 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d"] May 25 13:08:34.783826 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:34.783789 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeb4909_83b4_4b8a_ac96_d20a74970d44.slice/crio-9fe27b34aeaa3de2650db45db2b35c442fd3e7b908fe3a467a335f13ab935fdf WatchSource:0}: Error finding container 9fe27b34aeaa3de2650db45db2b35c442fd3e7b908fe3a467a335f13ab935fdf: Status 404 returned error can't find the container with id 9fe27b34aeaa3de2650db45db2b35c442fd3e7b908fe3a467a335f13ab935fdf May 25 13:08:35.251767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:35.251727 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg88x" event={"ID":"17da813c-56ef-431e-906e-02d26698b3ab","Type":"ContainerStarted","Data":"98fc47f493f416d9cd998b926a7d94dc21e50f37ebe6931bb3be75ef9106057f"} May 25 13:08:35.252705 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:35.252674 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" event={"ID":"6aeb4909-83b4-4b8a-ac96-d20a74970d44","Type":"ContainerStarted","Data":"9fe27b34aeaa3de2650db45db2b35c442fd3e7b908fe3a467a335f13ab935fdf"} May 25 13:08:35.269566 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:35.269522 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tg88x" podStartSLOduration=3.533172405 podStartE2EDuration="5.269508692s" podCreationTimestamp="2026-05-25 13:08:30 +0000 UTC" firstStartedPulling="2026-05-25 13:08:32.582185272 +0000 UTC m=+124.260482856" lastFinishedPulling="2026-05-25 13:08:34.318521562 +0000 UTC m=+125.996819143" observedRunningTime="2026-05-25 13:08:35.268798624 +0000 UTC m=+126.947096227" watchObservedRunningTime="2026-05-25 13:08:35.269508692 +0000 UTC m=+126.947806293" May 25 13:08:36.794755 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:36.794683 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64"] May 25 13:08:36.798675 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:36.798657 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:36.800993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:36.800971 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" May 25 13:08:36.801108 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:36.801005 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d6cqb\"" May 25 13:08:36.808071 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:36.808049 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64"] May 25 13:08:36.899518 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:36.899496 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/188626a6-0721-460f-9937-a93d4413732f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p8g64\" (UID: \"188626a6-0721-460f-9937-a93d4413732f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:37.000780 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:37.000753 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/188626a6-0721-460f-9937-a93d4413732f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p8g64\" (UID: \"188626a6-0721-460f-9937-a93d4413732f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:37.000897 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:37.000881 2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found May 25 13:08:37.000952 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:37.000936 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/188626a6-0721-460f-9937-a93d4413732f-tls-certificates podName:188626a6-0721-460f-9937-a93d4413732f nodeName:}" failed. No retries permitted until 2026-05-25 13:08:37.50092233 +0000 UTC m=+129.179219922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/188626a6-0721-460f-9937-a93d4413732f-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" (UID: "188626a6-0721-460f-9937-a93d4413732f") : secret "prometheus-operator-admission-webhook-tls" not found May 25 13:08:37.262594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:37.262559 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" event={"ID":"6aeb4909-83b4-4b8a-ac96-d20a74970d44","Type":"ContainerStarted","Data":"d573a41fd497e6a751491fb7d3fe385e79649cf2e70c4a9f6932ca3059909b89"} May 25 13:08:37.279998 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:37.279954 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-zb92d" podStartSLOduration=33.743474263 podStartE2EDuration="35.279943638s" podCreationTimestamp="2026-05-25 13:08:02 +0000 UTC" firstStartedPulling="2026-05-25 13:08:34.785654694 +0000 UTC m=+126.463952278" lastFinishedPulling="2026-05-25 13:08:36.322124061 +0000 UTC m=+128.000421653" observedRunningTime="2026-05-25 13:08:37.279029401 +0000 UTC m=+128.957327018" watchObservedRunningTime="2026-05-25 13:08:37.279943638 +0000 UTC m=+128.958241252" May 25 13:08:37.505082 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:37.505057 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/188626a6-0721-460f-9937-a93d4413732f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p8g64\" (UID: \"188626a6-0721-460f-9937-a93d4413732f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:37.507413 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:37.507384 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/188626a6-0721-460f-9937-a93d4413732f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-p8g64\" (UID: \"188626a6-0721-460f-9937-a93d4413732f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:37.707557 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:37.707532 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:37.828153 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:37.828126 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64"] May 25 13:08:37.830174 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:37.830148 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188626a6_0721_460f_9937_a93d4413732f.slice/crio-1b9f4289e487183bda41c84e3e4a79bdf338449294d0813fe3eee02557e59020 WatchSource:0}: Error finding container 1b9f4289e487183bda41c84e3e4a79bdf338449294d0813fe3eee02557e59020: Status 404 returned error can't find the container with id 1b9f4289e487183bda41c84e3e4a79bdf338449294d0813fe3eee02557e59020 May 25 13:08:38.265751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:38.265713 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" event={"ID":"188626a6-0721-460f-9937-a93d4413732f","Type":"ContainerStarted","Data":"1b9f4289e487183bda41c84e3e4a79bdf338449294d0813fe3eee02557e59020"} May 25 13:08:38.713554 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:38.713516 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:08:38.715912 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:38.715890 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b6fca-f227-41c5-9d85-6c3b82019d24-metrics-certs\") pod \"network-metrics-daemon-2228g\" (UID: \"1e2b6fca-f227-41c5-9d85-6c3b82019d24\") " pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:08:38.913775 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:38.913746 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xxkxl\"" May 25 13:08:38.921818 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:38.921797 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2228g" May 25 13:08:39.306908 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:39.306883 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2228g"] May 25 13:08:39.309204 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:39.309180 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2b6fca_f227_41c5_9d85_6c3b82019d24.slice/crio-8957f914496a8399ace231b1e259df046967dec529010dd471cee3804966eb7c WatchSource:0}: Error finding container 8957f914496a8399ace231b1e259df046967dec529010dd471cee3804966eb7c: Status 404 returned error can't find the container with id 8957f914496a8399ace231b1e259df046967dec529010dd471cee3804966eb7c May 25 13:08:40.273411 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:40.273380 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2228g" event={"ID":"1e2b6fca-f227-41c5-9d85-6c3b82019d24","Type":"ContainerStarted","Data":"8957f914496a8399ace231b1e259df046967dec529010dd471cee3804966eb7c"} May 25 13:08:40.274505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:40.274478 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" event={"ID":"188626a6-0721-460f-9937-a93d4413732f","Type":"ContainerStarted","Data":"b5ffdfe500b138f0db94c97494c72535d43a4e6ec3eea2e3c15d458d2e2c682e"} May 25 13:08:40.274698 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:40.274678 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:40.279873 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:40.279855 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" May 25 13:08:40.290262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:40.290218 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-p8g64" podStartSLOduration=2.587744211 podStartE2EDuration="4.290204355s" podCreationTimestamp="2026-05-25 13:08:36 +0000 UTC" firstStartedPulling="2026-05-25 13:08:37.832028624 +0000 UTC m=+129.510326207" lastFinishedPulling="2026-05-25 13:08:39.534488757 +0000 UTC m=+131.212786351" observedRunningTime="2026-05-25 13:08:40.289955922 +0000 UTC m=+131.968253523" watchObservedRunningTime="2026-05-25 13:08:40.290204355 +0000 UTC m=+131.968501958" May 25 13:08:40.661848 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:40.661804 2554 patch_prober.go:28] interesting pod/image-registry-646f7c65f5-v9krd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} May 25 13:08:40.661973 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:40.661881 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" podUID="d8d7fa88-8808-4d44-b234-377a131ed4b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:08:41.279035 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:41.278948 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2228g" event={"ID":"1e2b6fca-f227-41c5-9d85-6c3b82019d24","Type":"ContainerStarted","Data":"c762837399cc99daa52ca2a5116e02a88e318840475de45ecbd7c40cf28366ee"} May 25 13:08:41.279035 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:41.278992 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2228g" event={"ID":"1e2b6fca-f227-41c5-9d85-6c3b82019d24","Type":"ContainerStarted","Data":"be98188c8fc4c32a5b6b4750ae3fd41a47ef9809e763c319d8783948b2b09bf9"} May 25 13:08:41.296641 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:41.296596 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2228g" podStartSLOduration=130.933127116 podStartE2EDuration="2m12.296583151s" podCreationTimestamp="2026-05-25 13:06:29 +0000 UTC" firstStartedPulling="2026-05-25 13:08:39.311021234 +0000 UTC m=+130.989318817" lastFinishedPulling="2026-05-25 13:08:40.674477251 +0000 UTC m=+132.352774852" observedRunningTime="2026-05-25 13:08:41.296298699 +0000 UTC m=+132.974596301" watchObservedRunningTime="2026-05-25 13:08:41.296583151 +0000 UTC m=+132.974880752" May 25 13:08:45.271844 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.271814 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w72sz"] May 25 13:08:45.274000 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.273984 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.276045 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.276015 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 25 13:08:45.276873 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.276855 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 25 13:08:45.276993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.276902 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 25 13:08:45.276993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.276855 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 25 13:08:45.276993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.276855 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mwjdt\"" May 25 13:08:45.464468 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464414 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464498 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-root\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464562 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-sys\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464578 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08dc7950-719f-492b-8afe-e453b596a78d-metrics-client-ca\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-textfile\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464635 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464657 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6pd5\" (UniqueName: \"kubernetes.io/projected/08dc7950-719f-492b-8afe-e453b596a78d-kube-api-access-x6pd5\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464685 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-tls\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.464824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.464714 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-wtmp\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.565805 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.565744 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-textfile\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.565805 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.565777 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.565805 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.565796 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x6pd5\" (UniqueName: \"kubernetes.io/projected/08dc7950-719f-492b-8afe-e453b596a78d-kube-api-access-x6pd5\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566020 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.565994 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-tls\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566080 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566041 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-wtmp\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566141 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566125 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566188 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566157 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-root\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566241 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566200 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-sys\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566241 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566231 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-textfile\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566338 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566250 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-sys\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566338 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566260 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-wtmp\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566338 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566290 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08dc7950-719f-492b-8afe-e453b596a78d-metrics-client-ca\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566338 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566290 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08dc7950-719f-492b-8afe-e453b596a78d-root\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566566 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566385 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.566713 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.566691 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08dc7950-719f-492b-8afe-e453b596a78d-metrics-client-ca\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.568347 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.568307 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.568347 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.568343 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08dc7950-719f-492b-8afe-e453b596a78d-node-exporter-tls\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.573474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.573452 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6pd5\" (UniqueName: \"kubernetes.io/projected/08dc7950-719f-492b-8afe-e453b596a78d-kube-api-access-x6pd5\") pod \"node-exporter-w72sz\" (UID: \"08dc7950-719f-492b-8afe-e453b596a78d\") " pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.583369 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.583352 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w72sz" May 25 13:08:45.590958 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:45.590937 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dc7950_719f_492b_8afe_e453b596a78d.slice/crio-e73ab867b0088fba1a82d00c1886f028636bf2d275f26d6572c34c3507ef8850 WatchSource:0}: Error finding container e73ab867b0088fba1a82d00c1886f028636bf2d275f26d6572c34c3507ef8850: Status 404 returned error can't find the container with id e73ab867b0088fba1a82d00c1886f028636bf2d275f26d6572c34c3507ef8850 May 25 13:08:45.897076 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:45.897047 2554 scope.go:117] "RemoveContainer" containerID="7293e6649187cc2060c49908eb998920aa249c4ef3bfe70c86a61ee88001314e" May 25 13:08:45.897274 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:45.897253 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-jfbcj_openshift-console-operator(077160a2-2714-4cf9-961e-4169dc361eaa)\"" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" podUID="077160a2-2714-4cf9-961e-4169dc361eaa" May 25 13:08:46.296239 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.296162 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w72sz" event={"ID":"08dc7950-719f-492b-8afe-e453b596a78d","Type":"ContainerStarted","Data":"e73ab867b0088fba1a82d00c1886f028636bf2d275f26d6572c34c3507ef8850"} May 25 13:08:46.306991 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.306961 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:08:46.312222 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.312200 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.315053 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315029 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" May 25 13:08:46.315179 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315132 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" May 25 13:08:46.315354 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315335 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k4z9h\"" May 25 13:08:46.315548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315527 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" May 25 13:08:46.315847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315551 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" May 25 13:08:46.315847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315700 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" May 25 13:08:46.315847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315607 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" May 25 13:08:46.315847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315624 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" May 25 13:08:46.315847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315626 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" May 25 13:08:46.315847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.315384 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" May 25 13:08:46.325439 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.325398 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:08:46.373214 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373190 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373340 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373225 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373340 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373249 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373341 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373392 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373449 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vdj\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-kube-api-access-69vdj\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373600 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373510 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373600 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373578 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373701 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373641 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373701 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373700 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373723 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-out\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.373787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.373745 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-web-config\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474245 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474202 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474283 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474316 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474374 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474398 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474448 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-69vdj\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-kube-api-access-69vdj\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474497 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474527 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474567 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474599 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474614 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474638 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-out\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.474715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.474660 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-web-config\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.475346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.475237 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.475868 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:46.475803 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle podName:3c754e20-176e-4da0-b8d5-fd2664dd810c nodeName:}" failed. No retries permitted until 2026-05-25 13:08:46.975373098 +0000 UTC m=+138.653670686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c") : configmap references non-existent config key: ca-bundle.crt May 25 13:08:46.477018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.476684 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.477229 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.477172 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.477877 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.477837 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.478307 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.478093 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-web-config\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.478307 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.478255 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.478472 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.478320 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.478540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.478518 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-out\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.479075 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.479037 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.479949 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.479928 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.480200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.480185 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.483584 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.483563 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vdj\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-kube-api-access-69vdj\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.978016 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.977951 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:46.978764 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:46.978744 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:47.224752 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:47.224726 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 25 13:08:47.300419 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:47.300382 2554 generic.go:358] "Generic (PLEG): container finished" podID="08dc7950-719f-492b-8afe-e453b596a78d" containerID="3fd4b2704369954e31721f01f6ac49ffda380484e36c86fa6ba2b4da0f89f822" exitCode=0 May 25 13:08:47.300846 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:47.300462 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w72sz" event={"ID":"08dc7950-719f-492b-8afe-e453b596a78d","Type":"ContainerDied","Data":"3fd4b2704369954e31721f01f6ac49ffda380484e36c86fa6ba2b4da0f89f822"} May 25 13:08:47.363539 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:47.363499 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:08:47.365956 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:47.365931 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c754e20_176e_4da0_b8d5_fd2664dd810c.slice/crio-f4ff8898c48862f528136266fe1b4f1581bdbf27e037f7bc71473a466584f3bc WatchSource:0}: Error finding container f4ff8898c48862f528136266fe1b4f1581bdbf27e037f7bc71473a466584f3bc: Status 404 returned error can't find the container with id f4ff8898c48862f528136266fe1b4f1581bdbf27e037f7bc71473a466584f3bc May 25 13:08:48.305932 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:48.305892 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w72sz" event={"ID":"08dc7950-719f-492b-8afe-e453b596a78d","Type":"ContainerStarted","Data":"173143585ac64901ad616c9be71ddfebb8de2ba3862f495b3755276e6f73434a"} May 25 13:08:48.305932 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:48.305935 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w72sz" event={"ID":"08dc7950-719f-492b-8afe-e453b596a78d","Type":"ContainerStarted","Data":"1abe6225b4bd089442217a87904cd994869b762d4216ab42fc9350e009fd7128"} May 25 13:08:48.307072 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:48.307046 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerStarted","Data":"f4ff8898c48862f528136266fe1b4f1581bdbf27e037f7bc71473a466584f3bc"} May 25 13:08:48.328470 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:48.328412 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w72sz" podStartSLOduration=2.466934551 podStartE2EDuration="3.328400287s" podCreationTimestamp="2026-05-25 13:08:45 +0000 UTC" firstStartedPulling="2026-05-25 13:08:45.592699349 +0000 UTC m=+137.270996932" lastFinishedPulling="2026-05-25 13:08:46.454165087 +0000 UTC m=+138.132462668" observedRunningTime="2026-05-25 13:08:48.326508174 +0000 UTC m=+140.004805775" watchObservedRunningTime="2026-05-25 13:08:48.328400287 +0000 UTC m=+140.006697929" May 25 13:08:49.311077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:49.311042 2554 generic.go:358] "Generic (PLEG): container finished" podID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerID="8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8" exitCode=0 May 25 13:08:49.311420 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:49.311123 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8"} May 25 13:08:50.011851 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.011816 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg"] May 25 13:08:50.018913 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.018897 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:50.021019 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.021000 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" May 25 13:08:50.021099 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.021084 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-r4z28\"" May 25 13:08:50.026386 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.026360 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg"] May 25 13:08:50.103198 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.103171 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f235c12c-ecc9-4199-9343-bdeb4dc50486-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-cnrrg\" (UID: \"f235c12c-ecc9-4199-9343-bdeb4dc50486\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:50.203640 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.203604 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f235c12c-ecc9-4199-9343-bdeb4dc50486-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-cnrrg\" (UID: \"f235c12c-ecc9-4199-9343-bdeb4dc50486\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:50.203757 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:50.203720 2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found May 25 13:08:50.203813 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:50.203779 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f235c12c-ecc9-4199-9343-bdeb4dc50486-monitoring-plugin-cert podName:f235c12c-ecc9-4199-9343-bdeb4dc50486 nodeName:}" failed. No retries permitted until 2026-05-25 13:08:50.703765088 +0000 UTC m=+142.382062667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f235c12c-ecc9-4199-9343-bdeb4dc50486-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-cnrrg" (UID: "f235c12c-ecc9-4199-9343-bdeb4dc50486") : secret "monitoring-plugin-cert" not found May 25 13:08:50.661925 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.661892 2554 patch_prober.go:28] interesting pod/image-registry-646f7c65f5-v9krd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} May 25 13:08:50.662360 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.661947 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" podUID="d8d7fa88-8808-4d44-b234-377a131ed4b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:08:50.707949 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.707920 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f235c12c-ecc9-4199-9343-bdeb4dc50486-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-cnrrg\" (UID: \"f235c12c-ecc9-4199-9343-bdeb4dc50486\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:50.710501 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.710475 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f235c12c-ecc9-4199-9343-bdeb4dc50486-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-cnrrg\" (UID: \"f235c12c-ecc9-4199-9343-bdeb4dc50486\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:50.928253 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:50.928164 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:51.125109 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:51.125085 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg"] May 25 13:08:51.128467 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:51.128437 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf235c12c_ecc9_4199_9343_bdeb4dc50486.slice/crio-6c9782bc5745d2f523b5887da201dfa6b07c473ee7b5dbf0052a1dc1a681f38b WatchSource:0}: Error finding container 6c9782bc5745d2f523b5887da201dfa6b07c473ee7b5dbf0052a1dc1a681f38b: Status 404 returned error can't find the container with id 6c9782bc5745d2f523b5887da201dfa6b07c473ee7b5dbf0052a1dc1a681f38b May 25 13:08:51.317316 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:51.317288 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" event={"ID":"f235c12c-ecc9-4199-9343-bdeb4dc50486","Type":"ContainerStarted","Data":"6c9782bc5745d2f523b5887da201dfa6b07c473ee7b5dbf0052a1dc1a681f38b"} May 25 13:08:51.320228 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:51.320206 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerStarted","Data":"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5"} May 25 13:08:51.320318 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:51.320235 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerStarted","Data":"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3"} May 25 13:08:51.320318 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:51.320250 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerStarted","Data":"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6"} May 25 13:08:51.320318 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:51.320261 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerStarted","Data":"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2"} May 25 13:08:51.320318 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:51.320274 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerStarted","Data":"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac"} May 25 13:08:52.243785 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:52.243760 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78f78dfc48-kpjv4" May 25 13:08:52.328888 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:52.328859 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerStarted","Data":"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244"} May 25 13:08:52.355795 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:52.355752 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.695769867 podStartE2EDuration="6.355738048s" podCreationTimestamp="2026-05-25 13:08:46 +0000 UTC" firstStartedPulling="2026-05-25 13:08:47.367895304 +0000 UTC m=+139.046192891" lastFinishedPulling="2026-05-25 13:08:52.027863491 +0000 UTC m=+143.706161072" observedRunningTime="2026-05-25 13:08:52.354009237 +0000 UTC m=+144.032306860" watchObservedRunningTime="2026-05-25 13:08:52.355738048 +0000 UTC m=+144.034035649" May 25 13:08:53.333369 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:53.333334 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" event={"ID":"f235c12c-ecc9-4199-9343-bdeb4dc50486","Type":"ContainerStarted","Data":"b417e829dce6f25e79c95d0cf21d646581892e122d22f274f182c33fc31e13f1"} May 25 13:08:53.333803 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:53.333457 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:53.338267 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:53.338244 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" May 25 13:08:53.349598 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:53.349560 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-cnrrg" podStartSLOduration=3.018586935 podStartE2EDuration="4.349544874s" podCreationTimestamp="2026-05-25 13:08:49 +0000 UTC" firstStartedPulling="2026-05-25 13:08:51.130634743 +0000 UTC m=+142.808932342" lastFinishedPulling="2026-05-25 13:08:52.461592697 +0000 UTC m=+144.139890281" observedRunningTime="2026-05-25 13:08:53.348564285 +0000 UTC m=+145.026861899" watchObservedRunningTime="2026-05-25 13:08:53.349544874 +0000 UTC m=+145.027842476" May 25 13:08:55.675515 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.675469 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" podUID="d8d7fa88-8808-4d44-b234-377a131ed4b3" containerName="registry" containerID="cri-o://c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222" gracePeriod=30 May 25 13:08:55.907323 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.907303 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:55.950992 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.950933 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.950992 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.950980 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-certificates\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.951132 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951004 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-image-registry-private-configuration\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.951132 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951118 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-bound-sa-token\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.951210 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951154 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-installation-pull-secrets\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.951210 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951202 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7fa88-8808-4d44-b234-377a131ed4b3-ca-trust-extracted\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.951319 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951230 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-trusted-ca\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.951319 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951293 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vhfk\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-kube-api-access-6vhfk\") pod \"d8d7fa88-8808-4d44-b234-377a131ed4b3\" (UID: \"d8d7fa88-8808-4d44-b234-377a131ed4b3\") " May 25 13:08:55.951414 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951346 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:08:55.952200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951603 2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-certificates\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:55.952200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.951826 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:08:55.953410 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.953351 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:08:55.953410 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.953396 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:08:55.953572 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.953514 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:08:55.953618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.953563 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:08:55.953848 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.953832 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-kube-api-access-6vhfk" (OuterVolumeSpecName: "kube-api-access-6vhfk") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "kube-api-access-6vhfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:08:55.959776 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:55.959753 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d7fa88-8808-4d44-b234-377a131ed4b3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d8d7fa88-8808-4d44-b234-377a131ed4b3" (UID: "d8d7fa88-8808-4d44-b234-377a131ed4b3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:08:56.052174 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.052152 2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-image-registry-private-configuration\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:56.052174 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.052172 2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-bound-sa-token\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:56.052299 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.052182 2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8d7fa88-8808-4d44-b234-377a131ed4b3-installation-pull-secrets\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:56.052299 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.052191 2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8d7fa88-8808-4d44-b234-377a131ed4b3-ca-trust-extracted\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:56.052299 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.052200 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8d7fa88-8808-4d44-b234-377a131ed4b3-trusted-ca\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:56.052299 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.052210 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhfk\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-kube-api-access-6vhfk\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:56.052299 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.052218 2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8d7fa88-8808-4d44-b234-377a131ed4b3-registry-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:08:56.343041 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.343006 2554 generic.go:358] "Generic (PLEG): container finished" podID="d8d7fa88-8808-4d44-b234-377a131ed4b3" containerID="c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222" exitCode=0 May 25 13:08:56.343167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.343073 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" May 25 13:08:56.343167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.343096 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" event={"ID":"d8d7fa88-8808-4d44-b234-377a131ed4b3","Type":"ContainerDied","Data":"c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222"} May 25 13:08:56.343167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.343136 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646f7c65f5-v9krd" event={"ID":"d8d7fa88-8808-4d44-b234-377a131ed4b3","Type":"ContainerDied","Data":"abea31cf257fd519d0202fcb6104df4c1a191d068191f45893d25224b16faada"} May 25 13:08:56.343167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.343152 2554 scope.go:117] "RemoveContainer" containerID="c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222" May 25 13:08:56.351285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.351174 2554 scope.go:117] "RemoveContainer" containerID="c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222" May 25 13:08:56.351514 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:08:56.351488 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222\": container with ID starting with c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222 not found: ID does not exist" containerID="c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222" May 25 13:08:56.351609 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.351526 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222"} err="failed to get container status \"c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222\": rpc error: code = NotFound desc = could not find container \"c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222\": container with ID starting with c19afe9d79c09593b13fd44b15a7c24c70efa2cc78f0d5c583b64734a4a0c222 not found: ID does not exist" May 25 13:08:56.366850 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.366828 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-646f7c65f5-v9krd"] May 25 13:08:56.370874 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.370856 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-646f7c65f5-v9krd"] May 25 13:08:56.900787 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:56.900757 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d7fa88-8808-4d44-b234-377a131ed4b3" path="/var/lib/kubelet/pods/d8d7fa88-8808-4d44-b234-377a131ed4b3/volumes" May 25 13:08:58.902299 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:58.902272 2554 scope.go:117] "RemoveContainer" containerID="7293e6649187cc2060c49908eb998920aa249c4ef3bfe70c86a61ee88001314e" May 25 13:08:59.354901 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.354868 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:08:59.355052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.354958 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" event={"ID":"077160a2-2714-4cf9-961e-4169dc361eaa","Type":"ContainerStarted","Data":"6b0b142732ddacf88e1458a864275d975f2c848376bef1d4a91119a6a6d50b51"} May 25 13:08:59.355231 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.355208 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:59.365079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.365049 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" May 25 13:08:59.377014 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.376962 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-jfbcj" podStartSLOduration=53.668156644 podStartE2EDuration="57.376950028s" podCreationTimestamp="2026-05-25 13:08:02 +0000 UTC" firstStartedPulling="2026-05-25 13:08:03.009665718 +0000 UTC m=+94.687963298" lastFinishedPulling="2026-05-25 13:08:06.718459097 +0000 UTC m=+98.396756682" observedRunningTime="2026-05-25 13:08:59.375708074 +0000 UTC m=+151.054005677" watchObservedRunningTime="2026-05-25 13:08:59.376950028 +0000 UTC m=+151.055247630" May 25 13:08:59.399446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.399401 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-9sgf6"] May 25 13:08:59.399753 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.399736 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d7fa88-8808-4d44-b234-377a131ed4b3" containerName="registry" May 25 13:08:59.399833 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.399755 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d7fa88-8808-4d44-b234-377a131ed4b3" containerName="registry" May 25 13:08:59.399833 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.399822 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="d8d7fa88-8808-4d44-b234-377a131ed4b3" containerName="registry" May 25 13:08:59.401613 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.401595 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-9sgf6" May 25 13:08:59.404199 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.404178 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ckdtq\"" May 25 13:08:59.404312 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.404218 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" May 25 13:08:59.404699 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.404565 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" May 25 13:08:59.411351 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.411328 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-9sgf6"] May 25 13:08:59.478219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.478198 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd57z\" (UniqueName: \"kubernetes.io/projected/5f52192a-99b2-4f70-be41-8a4d7037f25f-kube-api-access-bd57z\") pod \"downloads-57bd96b888-9sgf6\" (UID: \"5f52192a-99b2-4f70-be41-8a4d7037f25f\") " pod="openshift-console/downloads-57bd96b888-9sgf6" May 25 13:08:59.578677 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.578648 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bd57z\" (UniqueName: \"kubernetes.io/projected/5f52192a-99b2-4f70-be41-8a4d7037f25f-kube-api-access-bd57z\") pod \"downloads-57bd96b888-9sgf6\" (UID: \"5f52192a-99b2-4f70-be41-8a4d7037f25f\") " pod="openshift-console/downloads-57bd96b888-9sgf6" May 25 13:08:59.586634 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.586614 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd57z\" (UniqueName: \"kubernetes.io/projected/5f52192a-99b2-4f70-be41-8a4d7037f25f-kube-api-access-bd57z\") pod \"downloads-57bd96b888-9sgf6\" (UID: \"5f52192a-99b2-4f70-be41-8a4d7037f25f\") " pod="openshift-console/downloads-57bd96b888-9sgf6" May 25 13:08:59.712498 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.712444 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-9sgf6" May 25 13:08:59.832373 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:08:59.832132 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-9sgf6"] May 25 13:08:59.834908 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:08:59.834873 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f52192a_99b2_4f70_be41_8a4d7037f25f.slice/crio-bc2ea5ac4a77f31d690a59fab97799720661ee8da58f835e6dd21ac58170bb09 WatchSource:0}: Error finding container bc2ea5ac4a77f31d690a59fab97799720661ee8da58f835e6dd21ac58170bb09: Status 404 returned error can't find the container with id bc2ea5ac4a77f31d690a59fab97799720661ee8da58f835e6dd21ac58170bb09 May 25 13:09:00.359571 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:00.359534 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-9sgf6" event={"ID":"5f52192a-99b2-4f70-be41-8a4d7037f25f","Type":"ContainerStarted","Data":"bc2ea5ac4a77f31d690a59fab97799720661ee8da58f835e6dd21ac58170bb09"} May 25 13:09:05.189835 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:09:05.189792 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-h8xqz" podUID="a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8" May 25 13:09:05.203260 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:09:05.203221 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-64d4d" podUID="232bc31b-feac-4e2b-9fb3-f1d3c3a0a075" May 25 13:09:05.376305 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:05.376273 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:09:05.376482 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:05.376287 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h8xqz" May 25 13:09:09.125809 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.125769 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-77f9699f4d-8h4tt"] May 25 13:09:09.129047 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.129028 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.131640 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.131617 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" May 25 13:09:09.131780 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.131644 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" May 25 13:09:09.131780 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.131727 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" May 25 13:09:09.131884 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.131867 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" May 25 13:09:09.131938 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.131867 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5n9sh\"" May 25 13:09:09.132073 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.132001 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" May 25 13:09:09.144771 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.144743 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f9699f4d-8h4tt"] May 25 13:09:09.263638 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.263607 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-serving-cert\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.263790 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.263670 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-service-ca\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.263790 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.263699 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-oauth-config\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.263790 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.263728 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r667\" (UniqueName: \"kubernetes.io/projected/6c73b215-c84f-4b65-885a-0219e37a9f65-kube-api-access-9r667\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.263790 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.263748 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-oauth-serving-cert\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.263991 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.263828 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-console-config\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.365238 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.365209 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-serving-cert\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.365446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.365267 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-service-ca\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.365446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.365289 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-oauth-config\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.365446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.365315 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9r667\" (UniqueName: \"kubernetes.io/projected/6c73b215-c84f-4b65-885a-0219e37a9f65-kube-api-access-9r667\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.365446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.365342 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-oauth-serving-cert\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.365446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.365365 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-console-config\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.366177 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.366128 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-console-config\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.366380 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.366326 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-service-ca\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.366380 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.366343 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-oauth-serving-cert\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.368221 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.368201 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-oauth-config\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.368315 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.368260 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-serving-cert\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.374977 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.374957 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r667\" (UniqueName: \"kubernetes.io/projected/6c73b215-c84f-4b65-885a-0219e37a9f65-kube-api-access-9r667\") pod \"console-77f9699f4d-8h4tt\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:09.440389 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:09.440313 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:10.071840 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.071795 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:09:10.072015 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.071902 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:09:10.074510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.074484 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8-metrics-tls\") pod \"dns-default-h8xqz\" (UID: \"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8\") " pod="openshift-dns/dns-default-h8xqz" May 25 13:09:10.074651 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.074592 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/232bc31b-feac-4e2b-9fb3-f1d3c3a0a075-cert\") pod \"ingress-canary-64d4d\" (UID: \"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075\") " pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:09:10.179679 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.179649 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dnmsw\"" May 25 13:09:10.180217 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.180199 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6hqsn\"" May 25 13:09:10.188099 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.188075 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h8xqz" May 25 13:09:10.188232 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:10.188179 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-64d4d" May 25 13:09:15.372447 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:15.372399 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h8xqz"] May 25 13:09:15.378189 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:09:15.377929 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c708f1_3e3a_49c2_a692_9f1cf2eb36e8.slice/crio-dacf394896474dba2f7a3ca664fc804f3849ed212f9011cdb623ba4c24f47975 WatchSource:0}: Error finding container dacf394896474dba2f7a3ca664fc804f3849ed212f9011cdb623ba4c24f47975: Status 404 returned error can't find the container with id dacf394896474dba2f7a3ca664fc804f3849ed212f9011cdb623ba4c24f47975 May 25 13:09:15.407675 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:15.407648 2554 generic.go:358] "Generic (PLEG): container finished" podID="69d027ad-c810-423a-98fa-54547bad1795" containerID="ddff919abaa53c7fc96063c68e6d5ce82b11b2dcb4bd3d3a99476312b8f394e0" exitCode=0 May 25 13:09:15.407764 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:15.407717 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" event={"ID":"69d027ad-c810-423a-98fa-54547bad1795","Type":"ContainerDied","Data":"ddff919abaa53c7fc96063c68e6d5ce82b11b2dcb4bd3d3a99476312b8f394e0"} May 25 13:09:15.408092 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:15.408075 2554 scope.go:117] "RemoveContainer" containerID="ddff919abaa53c7fc96063c68e6d5ce82b11b2dcb4bd3d3a99476312b8f394e0" May 25 13:09:15.408815 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:15.408795 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8xqz" event={"ID":"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8","Type":"ContainerStarted","Data":"dacf394896474dba2f7a3ca664fc804f3849ed212f9011cdb623ba4c24f47975"} May 25 13:09:15.600549 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:15.600502 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-64d4d"] May 25 13:09:15.603620 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:15.603594 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f9699f4d-8h4tt"] May 25 13:09:15.604473 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:09:15.604441 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232bc31b_feac_4e2b_9fb3_f1d3c3a0a075.slice/crio-0319e620c3388ef98b55feb0b9209eddce417fc69b048f00c40e74937871ee15 WatchSource:0}: Error finding container 0319e620c3388ef98b55feb0b9209eddce417fc69b048f00c40e74937871ee15: Status 404 returned error can't find the container with id 0319e620c3388ef98b55feb0b9209eddce417fc69b048f00c40e74937871ee15 May 25 13:09:15.606054 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:09:15.606030 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c73b215_c84f_4b65_885a_0219e37a9f65.slice/crio-a449079cbd83002b3fcb38821ba7ab44a7565fcdbbea0892ab80db40ea20559b WatchSource:0}: Error finding container a449079cbd83002b3fcb38821ba7ab44a7565fcdbbea0892ab80db40ea20559b: Status 404 returned error can't find the container with id a449079cbd83002b3fcb38821ba7ab44a7565fcdbbea0892ab80db40ea20559b May 25 13:09:16.415298 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:16.415258 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-vt6mr" event={"ID":"69d027ad-c810-423a-98fa-54547bad1795","Type":"ContainerStarted","Data":"0826fdd577ff607e4e7e650a78890aeeb460b55f571d7f08495451354f951ecc"} May 25 13:09:16.417467 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:16.417391 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9699f4d-8h4tt" event={"ID":"6c73b215-c84f-4b65-885a-0219e37a9f65","Type":"ContainerStarted","Data":"a449079cbd83002b3fcb38821ba7ab44a7565fcdbbea0892ab80db40ea20559b"} May 25 13:09:16.419032 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:16.419006 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-9sgf6" event={"ID":"5f52192a-99b2-4f70-be41-8a4d7037f25f","Type":"ContainerStarted","Data":"42e15d4b21bf2a47d568d5d0bf8bc58f3b2886170e124fb1ca999b50a47a31f7"} May 25 13:09:16.419711 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:16.419689 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-9sgf6" May 25 13:09:16.422928 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:16.422906 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-64d4d" event={"ID":"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075","Type":"ContainerStarted","Data":"0319e620c3388ef98b55feb0b9209eddce417fc69b048f00c40e74937871ee15"} May 25 13:09:16.431494 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:16.431473 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-9sgf6" May 25 13:09:17.512496 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.509495 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-9sgf6" podStartSLOduration=3.023907102 podStartE2EDuration="18.509474008s" podCreationTimestamp="2026-05-25 13:08:59 +0000 UTC" firstStartedPulling="2026-05-25 13:08:59.836962107 +0000 UTC m=+151.515259687" lastFinishedPulling="2026-05-25 13:09:15.322528999 +0000 UTC m=+167.000826593" observedRunningTime="2026-05-25 13:09:16.459474358 +0000 UTC m=+168.137771960" watchObservedRunningTime="2026-05-25 13:09:17.509474008 +0000 UTC m=+169.187771613" May 25 13:09:17.512496 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.511155 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5489cdd44b-kfmlp"] May 25 13:09:17.535522 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.535223 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5489cdd44b-kfmlp"] May 25 13:09:17.535522 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.535354 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.544901 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.544829 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" May 25 13:09:17.640805 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.640690 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-serving-cert\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.640805 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.640748 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbxk\" (UniqueName: \"kubernetes.io/projected/c38e49d4-c25c-4273-aa8b-c73912c1e43f-kube-api-access-wnbxk\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.641036 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.640812 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-oauth-config\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.641036 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.640861 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-trusted-ca-bundle\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.641036 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.640900 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-config\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.641036 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.640924 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-service-ca\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.641036 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.640948 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-oauth-serving-cert\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.742129 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.742012 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-serving-cert\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.742129 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.742073 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbxk\" (UniqueName: \"kubernetes.io/projected/c38e49d4-c25c-4273-aa8b-c73912c1e43f-kube-api-access-wnbxk\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.742129 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.742102 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-oauth-config\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.742735 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.742137 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-trusted-ca-bundle\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.742735 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.742176 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-config\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.742735 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.742196 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-service-ca\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.742735 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.742213 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-oauth-serving-cert\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.743440 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.743368 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-service-ca\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.743440 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.743377 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-oauth-serving-cert\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.743644 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.743574 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-config\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.744568 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.744524 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-trusted-ca-bundle\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.747973 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.747927 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-serving-cert\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.749631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.749589 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-oauth-config\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.751897 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.751859 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbxk\" (UniqueName: \"kubernetes.io/projected/c38e49d4-c25c-4273-aa8b-c73912c1e43f-kube-api-access-wnbxk\") pod \"console-5489cdd44b-kfmlp\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:17.851586 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:17.851217 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:20.477392 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:20.477362 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5489cdd44b-kfmlp"] May 25 13:09:21.453021 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.452979 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9699f4d-8h4tt" event={"ID":"6c73b215-c84f-4b65-885a-0219e37a9f65","Type":"ContainerStarted","Data":"56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9"} May 25 13:09:21.454953 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.454920 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8xqz" event={"ID":"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8","Type":"ContainerStarted","Data":"8fb162e9c76a5ff6ac2cbdbc873ce2ee47a467bc0b9e6d106d50d56aa40ccaa4"} May 25 13:09:21.454953 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.454950 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8xqz" event={"ID":"a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8","Type":"ContainerStarted","Data":"9bd12568ee306caa447311599ddffda119183b686a128dddd1b0917a7626476f"} May 25 13:09:21.455142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.455065 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h8xqz" May 25 13:09:21.456481 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.456459 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-64d4d" event={"ID":"232bc31b-feac-4e2b-9fb3-f1d3c3a0a075","Type":"ContainerStarted","Data":"7f9a7fb60f2ec1a111fe407b24dc2d18db04d87ca6eac1473ab5ceb43bfbf8e3"} May 25 13:09:21.458142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.458117 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5489cdd44b-kfmlp" event={"ID":"c38e49d4-c25c-4273-aa8b-c73912c1e43f","Type":"ContainerStarted","Data":"355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8"} May 25 13:09:21.458244 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.458150 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5489cdd44b-kfmlp" event={"ID":"c38e49d4-c25c-4273-aa8b-c73912c1e43f","Type":"ContainerStarted","Data":"346f811cb79beee1e93cd9c9b99d7b60e9ccdd4a9f694f4f033a1ce5e6388820"} May 25 13:09:21.472843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.472805 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-77f9699f4d-8h4tt" podStartSLOduration=7.375520775 podStartE2EDuration="12.472793599s" podCreationTimestamp="2026-05-25 13:09:09 +0000 UTC" firstStartedPulling="2026-05-25 13:09:15.608642662 +0000 UTC m=+167.286940241" lastFinishedPulling="2026-05-25 13:09:20.705915482 +0000 UTC m=+172.384213065" observedRunningTime="2026-05-25 13:09:21.470824754 +0000 UTC m=+173.149122355" watchObservedRunningTime="2026-05-25 13:09:21.472793599 +0000 UTC m=+173.151091200" May 25 13:09:21.486205 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.486164 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-64d4d" podStartSLOduration=134.768347678 podStartE2EDuration="2m19.486152219s" podCreationTimestamp="2026-05-25 13:07:02 +0000 UTC" firstStartedPulling="2026-05-25 13:09:15.606735461 +0000 UTC m=+167.285033044" lastFinishedPulling="2026-05-25 13:09:20.324539996 +0000 UTC m=+172.002837585" observedRunningTime="2026-05-25 13:09:21.485049307 +0000 UTC m=+173.163346908" watchObservedRunningTime="2026-05-25 13:09:21.486152219 +0000 UTC m=+173.164449823" May 25 13:09:21.502558 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.502509 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-h8xqz" podStartSLOduration=134.562925378 podStartE2EDuration="2m19.502494538s" podCreationTimestamp="2026-05-25 13:07:02 +0000 UTC" firstStartedPulling="2026-05-25 13:09:15.380215669 +0000 UTC m=+167.058513265" lastFinishedPulling="2026-05-25 13:09:20.319784823 +0000 UTC m=+171.998082425" observedRunningTime="2026-05-25 13:09:21.500704739 +0000 UTC m=+173.179002342" watchObservedRunningTime="2026-05-25 13:09:21.502494538 +0000 UTC m=+173.180792143" May 25 13:09:21.518744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:21.518699 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5489cdd44b-kfmlp" podStartSLOduration=3.945887332 podStartE2EDuration="4.518684256s" podCreationTimestamp="2026-05-25 13:09:17 +0000 UTC" firstStartedPulling="2026-05-25 13:09:20.485932805 +0000 UTC m=+172.164230385" lastFinishedPulling="2026-05-25 13:09:21.058729719 +0000 UTC m=+172.737027309" observedRunningTime="2026-05-25 13:09:21.516888398 +0000 UTC m=+173.195186000" watchObservedRunningTime="2026-05-25 13:09:21.518684256 +0000 UTC m=+173.196981860" May 25 13:09:22.463286 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:22.463247 2554 generic.go:358] "Generic (PLEG): container finished" podID="720a0198-112a-4272-b799-e51dbaf56832" containerID="dce081fcf5af45759ddbf20f5d6ee914ff2757f9779575dbe5a3b51b5158fb3f" exitCode=0 May 25 13:09:22.463503 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:22.463332 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" event={"ID":"720a0198-112a-4272-b799-e51dbaf56832","Type":"ContainerDied","Data":"dce081fcf5af45759ddbf20f5d6ee914ff2757f9779575dbe5a3b51b5158fb3f"} May 25 13:09:22.464020 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:22.463994 2554 scope.go:117] "RemoveContainer" containerID="dce081fcf5af45759ddbf20f5d6ee914ff2757f9779575dbe5a3b51b5158fb3f" May 25 13:09:23.470135 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:23.470090 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-9669r" event={"ID":"720a0198-112a-4272-b799-e51dbaf56832","Type":"ContainerStarted","Data":"7e00d8825bb7cc08226b70b4c2e37ccf0e880dd11c38f574d6a61f934c1df823"} May 25 13:09:27.851656 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:27.851622 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:27.852116 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:27.851700 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:27.856306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:27.856279 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:28.489401 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:28.489370 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:09:28.536318 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:28.536282 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f9699f4d-8h4tt"] May 25 13:09:29.441039 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:29.441008 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:31.465150 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:31.465120 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h8xqz" May 25 13:09:32.500462 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:32.500415 2554 generic.go:358] "Generic (PLEG): container finished" podID="d5a29851-cc12-4af0-93df-a15c86f0ee86" containerID="4db3c753e577af0e9ded0d01dcd64fba61ae6cb9079e29dc1216b16ec714f4dd" exitCode=0 May 25 13:09:32.500822 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:32.500455 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" event={"ID":"d5a29851-cc12-4af0-93df-a15c86f0ee86","Type":"ContainerDied","Data":"4db3c753e577af0e9ded0d01dcd64fba61ae6cb9079e29dc1216b16ec714f4dd"} May 25 13:09:32.500887 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:32.500837 2554 scope.go:117] "RemoveContainer" containerID="4db3c753e577af0e9ded0d01dcd64fba61ae6cb9079e29dc1216b16ec714f4dd" May 25 13:09:33.505263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:33.505229 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-jc96l" event={"ID":"d5a29851-cc12-4af0-93df-a15c86f0ee86","Type":"ContainerStarted","Data":"201b78721ee95ccb2c2ec3cda227af859ed4e527e1e5da08792e2463020250c6"} May 25 13:09:53.555306 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:53.555244 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-77f9699f4d-8h4tt" podUID="6c73b215-c84f-4b65-885a-0219e37a9f65" containerName="console" containerID="cri-o://56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9" gracePeriod=15 May 25 13:09:53.845884 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:53.845862 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77f9699f4d-8h4tt_6c73b215-c84f-4b65-885a-0219e37a9f65/console/0.log" May 25 13:09:53.846001 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:53.845936 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:54.010275 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010241 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-oauth-serving-cert\") pod \"6c73b215-c84f-4b65-885a-0219e37a9f65\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " May 25 13:09:54.010461 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010292 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-oauth-config\") pod \"6c73b215-c84f-4b65-885a-0219e37a9f65\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " May 25 13:09:54.010461 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010322 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r667\" (UniqueName: \"kubernetes.io/projected/6c73b215-c84f-4b65-885a-0219e37a9f65-kube-api-access-9r667\") pod \"6c73b215-c84f-4b65-885a-0219e37a9f65\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " May 25 13:09:54.010461 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010357 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-console-config\") pod \"6c73b215-c84f-4b65-885a-0219e37a9f65\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " May 25 13:09:54.010461 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010381 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-serving-cert\") pod \"6c73b215-c84f-4b65-885a-0219e37a9f65\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " May 25 13:09:54.010461 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010415 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-service-ca\") pod \"6c73b215-c84f-4b65-885a-0219e37a9f65\" (UID: \"6c73b215-c84f-4b65-885a-0219e37a9f65\") " May 25 13:09:54.010801 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010685 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6c73b215-c84f-4b65-885a-0219e37a9f65" (UID: "6c73b215-c84f-4b65-885a-0219e37a9f65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:09:54.010801 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010779 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-console-config" (OuterVolumeSpecName: "console-config") pod "6c73b215-c84f-4b65-885a-0219e37a9f65" (UID: "6c73b215-c84f-4b65-885a-0219e37a9f65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:09:54.011018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.010918 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-service-ca" (OuterVolumeSpecName: "service-ca") pod "6c73b215-c84f-4b65-885a-0219e37a9f65" (UID: "6c73b215-c84f-4b65-885a-0219e37a9f65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:09:54.012743 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.012715 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6c73b215-c84f-4b65-885a-0219e37a9f65" (UID: "6c73b215-c84f-4b65-885a-0219e37a9f65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:09:54.012830 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.012759 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c73b215-c84f-4b65-885a-0219e37a9f65-kube-api-access-9r667" (OuterVolumeSpecName: "kube-api-access-9r667") pod "6c73b215-c84f-4b65-885a-0219e37a9f65" (UID: "6c73b215-c84f-4b65-885a-0219e37a9f65"). InnerVolumeSpecName "kube-api-access-9r667". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:09:54.012830 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.012790 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6c73b215-c84f-4b65-885a-0219e37a9f65" (UID: "6c73b215-c84f-4b65-885a-0219e37a9f65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:09:54.111400 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.111344 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-oauth-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:09:54.111400 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.111364 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-oauth-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:09:54.111400 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.111378 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r667\" (UniqueName: \"kubernetes.io/projected/6c73b215-c84f-4b65-885a-0219e37a9f65-kube-api-access-9r667\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:09:54.111400 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.111394 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-console-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:09:54.111591 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.111403 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c73b215-c84f-4b65-885a-0219e37a9f65-console-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:09:54.111591 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.111411 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c73b215-c84f-4b65-885a-0219e37a9f65-service-ca\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:09:54.572365 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.572328 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77f9699f4d-8h4tt_6c73b215-c84f-4b65-885a-0219e37a9f65/console/0.log" May 25 13:09:54.572890 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.572384 2554 generic.go:358] "Generic (PLEG): container finished" podID="6c73b215-c84f-4b65-885a-0219e37a9f65" containerID="56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9" exitCode=2 May 25 13:09:54.572890 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.572473 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9699f4d-8h4tt" May 25 13:09:54.572890 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.572471 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9699f4d-8h4tt" event={"ID":"6c73b215-c84f-4b65-885a-0219e37a9f65","Type":"ContainerDied","Data":"56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9"} May 25 13:09:54.572890 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.572586 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9699f4d-8h4tt" event={"ID":"6c73b215-c84f-4b65-885a-0219e37a9f65","Type":"ContainerDied","Data":"a449079cbd83002b3fcb38821ba7ab44a7565fcdbbea0892ab80db40ea20559b"} May 25 13:09:54.572890 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.572615 2554 scope.go:117] "RemoveContainer" containerID="56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9" May 25 13:09:54.581121 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.581100 2554 scope.go:117] "RemoveContainer" containerID="56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9" May 25 13:09:54.581397 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:09:54.581378 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9\": container with ID starting with 56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9 not found: ID does not exist" containerID="56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9" May 25 13:09:54.581492 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.581410 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9"} err="failed to get container status \"56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9\": rpc error: code = NotFound desc = could not find container \"56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9\": container with ID starting with 56da1ce825e421b3cac4317b546de77bb59c3d0a47eee583a869ecf6cd1559e9 not found: ID does not exist" May 25 13:09:54.592963 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.592941 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f9699f4d-8h4tt"] May 25 13:09:54.596560 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.596539 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77f9699f4d-8h4tt"] May 25 13:09:54.900171 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:09:54.900107 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c73b215-c84f-4b65-885a-0219e37a9f65" path="/var/lib/kubelet/pods/6c73b215-c84f-4b65-885a-0219e37a9f65/volumes" May 25 13:10:05.515021 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:05.514985 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:10:05.515507 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:05.515456 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="alertmanager" containerID="cri-o://d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac" gracePeriod=120 May 25 13:10:05.515594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:05.515486 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-metric" containerID="cri-o://d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5" gracePeriod=120 May 25 13:10:05.515594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:05.515538 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="prom-label-proxy" containerID="cri-o://2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244" gracePeriod=120 May 25 13:10:05.515594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:05.515565 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="config-reloader" containerID="cri-o://8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2" gracePeriod=120 May 25 13:10:05.515734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:05.515554 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-web" containerID="cri-o://dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6" gracePeriod=120 May 25 13:10:05.515734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:05.515653 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy" containerID="cri-o://833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3" gracePeriod=120 May 25 13:10:06.610802 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610777 2554 generic.go:358] "Generic (PLEG): container finished" podID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerID="2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244" exitCode=0 May 25 13:10:06.610802 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610801 2554 generic.go:358] "Generic (PLEG): container finished" podID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerID="833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3" exitCode=0 May 25 13:10:06.611101 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610809 2554 generic.go:358] "Generic (PLEG): container finished" podID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerID="8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2" exitCode=0 May 25 13:10:06.611101 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610815 2554 generic.go:358] "Generic (PLEG): container finished" podID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerID="d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac" exitCode=0 May 25 13:10:06.611101 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610846 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244"} May 25 13:10:06.611101 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610874 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3"} May 25 13:10:06.611101 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610886 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2"} May 25 13:10:06.611101 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.610895 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac"} May 25 13:10:06.777790 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.777770 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:06.806600 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806580 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69vdj\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-kube-api-access-69vdj\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.806733 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806609 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-volume\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.806733 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806642 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-main-db\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.806830 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806733 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-web\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.806830 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806802 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.806923 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806833 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.806923 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806883 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-cluster-tls-config\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.807028 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806921 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-web-config\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.807028 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806949 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-metrics-client-ca\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.807028 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806987 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-main-tls\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.807206 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.807028 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-out\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.807206 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.807060 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.807206 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.807083 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-tls-assets\") pod \"3c754e20-176e-4da0-b8d5-fd2664dd810c\" (UID: \"3c754e20-176e-4da0-b8d5-fd2664dd810c\") " May 25 13:10:06.807206 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.806945 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:10:06.807451 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.807370 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-main-db\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.808865 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.808543 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:10:06.809560 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.809519 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:10:06.812074 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.812036 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:06.812383 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.812322 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:06.812804 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.812716 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-volume" (OuterVolumeSpecName: "config-volume") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:06.812804 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.812780 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:06.812960 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.812900 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-out" (OuterVolumeSpecName: "config-out") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:10:06.813713 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.813688 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-kube-api-access-69vdj" (OuterVolumeSpecName: "kube-api-access-69vdj") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "kube-api-access-69vdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:10:06.814188 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.814162 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:10:06.814780 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.814746 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:06.817841 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.817802 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:06.826592 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.826569 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-web-config" (OuterVolumeSpecName: "web-config") pod "3c754e20-176e-4da0-b8d5-fd2664dd810c" (UID: "3c754e20-176e-4da0-b8d5-fd2664dd810c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:06.907685 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907666 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-metrics-client-ca\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907686 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-main-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907696 2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-out\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907706 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c754e20-176e-4da0-b8d5-fd2664dd810c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907715 2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-tls-assets\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907724 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69vdj\" (UniqueName: \"kubernetes.io/projected/3c754e20-176e-4da0-b8d5-fd2664dd810c-kube-api-access-69vdj\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907732 2554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-config-volume\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907743 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907751 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907760 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.907767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907770 2554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-cluster-tls-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:06.908067 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:06.907778 2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c754e20-176e-4da0-b8d5-fd2664dd810c-web-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:07.621562 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.621506 2554 generic.go:358] "Generic (PLEG): container finished" podID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerID="d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5" exitCode=0 May 25 13:10:07.621562 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.621555 2554 generic.go:358] "Generic (PLEG): container finished" podID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerID="dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6" exitCode=0 May 25 13:10:07.621980 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.621597 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5"} May 25 13:10:07.621980 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.621693 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6"} May 25 13:10:07.621980 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.621733 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c754e20-176e-4da0-b8d5-fd2664dd810c","Type":"ContainerDied","Data":"f4ff8898c48862f528136266fe1b4f1581bdbf27e037f7bc71473a466584f3bc"} May 25 13:10:07.621980 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.621750 2554 scope.go:117] "RemoveContainer" containerID="2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244" May 25 13:10:07.621980 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.621753 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.629651 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.629634 2554 scope.go:117] "RemoveContainer" containerID="d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5" May 25 13:10:07.635840 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.635823 2554 scope.go:117] "RemoveContainer" containerID="833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3" May 25 13:10:07.641721 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.641700 2554 scope.go:117] "RemoveContainer" containerID="dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6" May 25 13:10:07.645665 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.645640 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:10:07.648741 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.648724 2554 scope.go:117] "RemoveContainer" containerID="8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2" May 25 13:10:07.649605 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.649583 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:10:07.655483 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.655467 2554 scope.go:117] "RemoveContainer" containerID="d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac" May 25 13:10:07.661629 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.661613 2554 scope.go:117] "RemoveContainer" containerID="8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8" May 25 13:10:07.667782 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.667765 2554 scope.go:117] "RemoveContainer" containerID="2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244" May 25 13:10:07.668007 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:07.667988 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244\": container with ID starting with 2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244 not found: ID does not exist" containerID="2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244" May 25 13:10:07.668075 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668017 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244"} err="failed to get container status \"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244\": rpc error: code = NotFound desc = could not find container \"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244\": container with ID starting with 2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244 not found: ID does not exist" May 25 13:10:07.668075 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668041 2554 scope.go:117] "RemoveContainer" containerID="d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5" May 25 13:10:07.668279 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:07.668261 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5\": container with ID starting with d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5 not found: ID does not exist" containerID="d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5" May 25 13:10:07.668320 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668286 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5"} err="failed to get container status \"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5\": rpc error: code = NotFound desc = could not find container \"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5\": container with ID starting with d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5 not found: ID does not exist" May 25 13:10:07.668320 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668307 2554 scope.go:117] "RemoveContainer" containerID="833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3" May 25 13:10:07.668556 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:07.668540 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3\": container with ID starting with 833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3 not found: ID does not exist" containerID="833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3" May 25 13:10:07.668622 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668564 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3"} err="failed to get container status \"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3\": rpc error: code = NotFound desc = could not find container \"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3\": container with ID starting with 833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3 not found: ID does not exist" May 25 13:10:07.668622 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668585 2554 scope.go:117] "RemoveContainer" containerID="dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6" May 25 13:10:07.668803 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:07.668785 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6\": container with ID starting with dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6 not found: ID does not exist" containerID="dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6" May 25 13:10:07.668840 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668808 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6"} err="failed to get container status \"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6\": rpc error: code = NotFound desc = could not find container \"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6\": container with ID starting with dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6 not found: ID does not exist" May 25 13:10:07.668840 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.668821 2554 scope.go:117] "RemoveContainer" containerID="8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2" May 25 13:10:07.669031 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:07.669016 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2\": container with ID starting with 8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2 not found: ID does not exist" containerID="8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2" May 25 13:10:07.669090 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669037 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2"} err="failed to get container status \"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2\": rpc error: code = NotFound desc = could not find container \"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2\": container with ID starting with 8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2 not found: ID does not exist" May 25 13:10:07.669090 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669056 2554 scope.go:117] "RemoveContainer" containerID="d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac" May 25 13:10:07.669241 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:07.669227 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac\": container with ID starting with d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac not found: ID does not exist" containerID="d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac" May 25 13:10:07.669278 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669247 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac"} err="failed to get container status \"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac\": rpc error: code = NotFound desc = could not find container \"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac\": container with ID starting with d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac not found: ID does not exist" May 25 13:10:07.669278 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669260 2554 scope.go:117] "RemoveContainer" containerID="8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8" May 25 13:10:07.669419 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:07.669406 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8\": container with ID starting with 8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8 not found: ID does not exist" containerID="8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8" May 25 13:10:07.669518 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669440 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8"} err="failed to get container status \"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8\": rpc error: code = NotFound desc = could not find container \"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8\": container with ID starting with 8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8 not found: ID does not exist" May 25 13:10:07.669518 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669458 2554 scope.go:117] "RemoveContainer" containerID="2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244" May 25 13:10:07.669678 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669660 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244"} err="failed to get container status \"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244\": rpc error: code = NotFound desc = could not find container \"2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244\": container with ID starting with 2539b1e7414dee7f542e2438fd4276bb3c6e3efb1bc12287cc79c5b997851244 not found: ID does not exist" May 25 13:10:07.669743 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669680 2554 scope.go:117] "RemoveContainer" containerID="d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5" May 25 13:10:07.669920 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669903 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5"} err="failed to get container status \"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5\": rpc error: code = NotFound desc = could not find container \"d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5\": container with ID starting with d60d0b14afad9f561961127afbcb4657ec846d98fb395087fd838ffa0a3bb2c5 not found: ID does not exist" May 25 13:10:07.669970 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.669921 2554 scope.go:117] "RemoveContainer" containerID="833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3" May 25 13:10:07.670094 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670079 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3"} err="failed to get container status \"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3\": rpc error: code = NotFound desc = could not find container \"833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3\": container with ID starting with 833b8a552dedd5683fa349eb9450a9fc94d2ede3b824c682379de68e9db592a3 not found: ID does not exist" May 25 13:10:07.670139 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670094 2554 scope.go:117] "RemoveContainer" containerID="dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6" May 25 13:10:07.670269 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670256 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6"} err="failed to get container status \"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6\": rpc error: code = NotFound desc = could not find container \"dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6\": container with ID starting with dc58828bf137b8a2954814440b2194056e6daa98ca6a192829aa82216815b4a6 not found: ID does not exist" May 25 13:10:07.670310 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670271 2554 scope.go:117] "RemoveContainer" containerID="8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2" May 25 13:10:07.670446 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670413 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2"} err="failed to get container status \"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2\": rpc error: code = NotFound desc = could not find container \"8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2\": container with ID starting with 8db5e3e7cc80e18b182c283be3561608726d475c0277fcb84b18417b9d1967c2 not found: ID does not exist" May 25 13:10:07.670500 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670445 2554 scope.go:117] "RemoveContainer" containerID="d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac" May 25 13:10:07.670613 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670595 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac"} err="failed to get container status \"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac\": rpc error: code = NotFound desc = could not find container \"d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac\": container with ID starting with d8529285e1bedd2b0ef26cb8070f80d460690e869c63113a8ac9b9a6466e43ac not found: ID does not exist" May 25 13:10:07.670680 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670616 2554 scope.go:117] "RemoveContainer" containerID="8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8" May 25 13:10:07.670832 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.670816 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8"} err="failed to get container status \"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8\": rpc error: code = NotFound desc = could not find container \"8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8\": container with ID starting with 8edcb7054cf998935a20e9515a026daab6326a474f9beae5e78f75571c4911b8 not found: ID does not exist" May 25 13:10:07.676538 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676520 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:10:07.676810 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676796 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-metric" May 25 13:10:07.676858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676812 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-metric" May 25 13:10:07.676858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676823 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="init-config-reloader" May 25 13:10:07.676858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676829 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="init-config-reloader" May 25 13:10:07.676858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676844 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy" May 25 13:10:07.676858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676849 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy" May 25 13:10:07.676858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676856 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="alertmanager" May 25 13:10:07.676858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676860 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="alertmanager" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676870 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c73b215-c84f-4b65-885a-0219e37a9f65" containerName="console" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676878 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c73b215-c84f-4b65-885a-0219e37a9f65" containerName="console" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676886 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="config-reloader" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676891 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="config-reloader" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676901 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="prom-label-proxy" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676906 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="prom-label-proxy" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676912 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-web" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676917 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-web" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676965 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="config-reloader" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676973 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-web" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676980 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="alertmanager" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676986 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c73b215-c84f-4b65-885a-0219e37a9f65" containerName="console" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676992 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.676998 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="kube-rbac-proxy-metric" May 25 13:10:07.677052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.677004 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" containerName="prom-label-proxy" May 25 13:10:07.680401 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.680385 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.683213 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683193 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" May 25 13:10:07.683213 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683207 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k4z9h\"" May 25 13:10:07.683448 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683194 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" May 25 13:10:07.683448 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683193 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" May 25 13:10:07.683448 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683198 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" May 25 13:10:07.683448 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683392 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" May 25 13:10:07.683668 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683474 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" May 25 13:10:07.683668 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683583 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" May 25 13:10:07.683668 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.683622 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" May 25 13:10:07.690450 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.690413 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" May 25 13:10:07.696202 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.696182 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:10:07.712115 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712088 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712205 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712124 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712205 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712143 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0113e92-ebec-4a04-90c6-68078134db4b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712205 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0113e92-ebec-4a04-90c6-68078134db4b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712205 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712181 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0113e92-ebec-4a04-90c6-68078134db4b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712205 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712198 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lf52\" (UniqueName: \"kubernetes.io/projected/a0113e92-ebec-4a04-90c6-68078134db4b-kube-api-access-5lf52\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712219 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712297 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0113e92-ebec-4a04-90c6-68078134db4b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712366 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0113e92-ebec-4a04-90c6-68078134db4b-config-out\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712413 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-web-config\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712664 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712470 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.712664 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.712504 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813389 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813419 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0113e92-ebec-4a04-90c6-68078134db4b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813449 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0113e92-ebec-4a04-90c6-68078134db4b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813663 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813624 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0113e92-ebec-4a04-90c6-68078134db4b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813663 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813656 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5lf52\" (UniqueName: \"kubernetes.io/projected/a0113e92-ebec-4a04-90c6-68078134db4b-kube-api-access-5lf52\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813762 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813693 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813762 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813736 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813762 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0113e92-ebec-4a04-90c6-68078134db4b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813788 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0113e92-ebec-4a04-90c6-68078134db4b-config-out\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813826 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-web-config\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813856 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.813866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813862 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0113e92-ebec-4a04-90c6-68078134db4b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.814097 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813889 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.814097 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.813943 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.814627 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.814603 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0113e92-ebec-4a04-90c6-68078134db4b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.815736 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.815577 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0113e92-ebec-4a04-90c6-68078134db4b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.816331 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.816301 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.816459 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.816371 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0113e92-ebec-4a04-90c6-68078134db4b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.816894 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.816851 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.816968 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.816895 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.817015 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.816973 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-web-config\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.817015 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.816972 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.817416 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.817394 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.818026 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.817998 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0113e92-ebec-4a04-90c6-68078134db4b-config-out\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.818113 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.818096 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0113e92-ebec-4a04-90c6-68078134db4b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.822351 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.822327 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lf52\" (UniqueName: \"kubernetes.io/projected/a0113e92-ebec-4a04-90c6-68078134db4b-kube-api-access-5lf52\") pod \"alertmanager-main-0\" (UID: \"a0113e92-ebec-4a04-90c6-68078134db4b\") " pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:07.988931 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:07.988908 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 25 13:10:08.117559 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:08.117532 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 25 13:10:08.120166 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:10:08.120140 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0113e92_ebec_4a04_90c6_68078134db4b.slice/crio-61ac8adb86c2900538bcc0e9db466ea54507697d52cbbf3be6f964b8a8129bc3 WatchSource:0}: Error finding container 61ac8adb86c2900538bcc0e9db466ea54507697d52cbbf3be6f964b8a8129bc3: Status 404 returned error can't find the container with id 61ac8adb86c2900538bcc0e9db466ea54507697d52cbbf3be6f964b8a8129bc3 May 25 13:10:08.625897 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:08.625866 2554 generic.go:358] "Generic (PLEG): container finished" podID="a0113e92-ebec-4a04-90c6-68078134db4b" containerID="473a2b181503e519df18463b62149fc1975d968ccadbb43b2397838e62d0fa32" exitCode=0 May 25 13:10:08.626234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:08.625959 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerDied","Data":"473a2b181503e519df18463b62149fc1975d968ccadbb43b2397838e62d0fa32"} May 25 13:10:08.626234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:08.626003 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerStarted","Data":"61ac8adb86c2900538bcc0e9db466ea54507697d52cbbf3be6f964b8a8129bc3"} May 25 13:10:08.902775 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:08.902744 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c754e20-176e-4da0-b8d5-fd2664dd810c" path="/var/lib/kubelet/pods/3c754e20-176e-4da0-b8d5-fd2664dd810c/volumes" May 25 13:10:09.608009 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.607977 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75f9b66f-svp5b"] May 25 13:10:09.610162 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.610146 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.613411 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.613392 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" May 25 13:10:09.613995 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.613976 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" May 25 13:10:09.614077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.614062 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" May 25 13:10:09.614218 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.614205 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" May 25 13:10:09.614258 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.614220 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-k9c9b\"" May 25 13:10:09.614435 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.614414 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" May 25 13:10:09.620447 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.620414 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" May 25 13:10:09.629040 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.629017 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f9b66f-svp5b"] May 25 13:10:09.629960 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.629939 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-secret-telemeter-client\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.630066 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.629989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.630128 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.630060 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-federate-client-tls\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.630128 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.630092 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-metrics-client-ca\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.630230 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.630146 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-serving-certs-ca-bundle\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.630230 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.630171 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-telemeter-client-tls\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.630230 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.630198 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nv4t\" (UniqueName: \"kubernetes.io/projected/c0b3f96f-81f4-4063-a627-069319cc24a1-kube-api-access-5nv4t\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.630356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.630235 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.633092 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.633066 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerStarted","Data":"d886f7c17ea49363cb23a15d56b7f83c543d3cd6d31ebfd15d334223c2f51ad9"} May 25 13:10:09.633196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.633099 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerStarted","Data":"9917a6720472ffaba1a6d80c10eeb9a1917e6ca0cf009134a74cef71c3983bfc"} May 25 13:10:09.633196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.633114 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerStarted","Data":"6fa06f7579e2dc8a914ee4869659c0a9e758075e92a9d20e44e04ddf327a6cc1"} May 25 13:10:09.633196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.633124 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerStarted","Data":"ec177bbaa31f3e148e6390523bf8d25545f5edfca7021c7f66d743fec0e2026c"} May 25 13:10:09.633196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.633132 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerStarted","Data":"c2e52a85668bf085897a2615975efa5d71637633bb5c8c83eda414c2a85f2c73"} May 25 13:10:09.633196 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.633143 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0113e92-ebec-4a04-90c6-68078134db4b","Type":"ContainerStarted","Data":"0bc7697e76dd1bd01d41828832837ee74aff363dcbab9c666cc8114aaca1eef0"} May 25 13:10:09.687264 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.687220 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.687206512 podStartE2EDuration="2.687206512s" podCreationTimestamp="2026-05-25 13:10:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:10:09.685758951 +0000 UTC m=+221.364056553" watchObservedRunningTime="2026-05-25 13:10:09.687206512 +0000 UTC m=+221.365504114" May 25 13:10:09.731349 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731320 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-federate-client-tls\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.731349 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731351 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-metrics-client-ca\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.731533 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731464 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-serving-certs-ca-bundle\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.731533 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731491 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-telemeter-client-tls\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.731533 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731521 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5nv4t\" (UniqueName: \"kubernetes.io/projected/c0b3f96f-81f4-4063-a627-069319cc24a1-kube-api-access-5nv4t\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.731693 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731568 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.731693 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731658 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-secret-telemeter-client\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.731811 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.731790 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.732122 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.732098 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-metrics-client-ca\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.732238 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.732189 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-serving-certs-ca-bundle\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.732726 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.732699 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b3f96f-81f4-4063-a627-069319cc24a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.734542 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.734518 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-federate-client-tls\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.734642 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.734525 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-telemeter-client-tls\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.734642 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.734593 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.734861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.734841 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c0b3f96f-81f4-4063-a627-069319cc24a1-secret-telemeter-client\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.740696 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.740678 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nv4t\" (UniqueName: \"kubernetes.io/projected/c0b3f96f-81f4-4063-a627-069319cc24a1-kube-api-access-5nv4t\") pod \"telemeter-client-75f9b66f-svp5b\" (UID: \"c0b3f96f-81f4-4063-a627-069319cc24a1\") " pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:09.920214 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:09.920149 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" May 25 13:10:10.044028 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:10.044005 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f9b66f-svp5b"] May 25 13:10:10.046226 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:10:10.046123 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b3f96f_81f4_4063_a627_069319cc24a1.slice/crio-072c2dcc7334623e8cbb270e70f46d26042e92e48bc5462ea9251408d35e9458 WatchSource:0}: Error finding container 072c2dcc7334623e8cbb270e70f46d26042e92e48bc5462ea9251408d35e9458: Status 404 returned error can't find the container with id 072c2dcc7334623e8cbb270e70f46d26042e92e48bc5462ea9251408d35e9458 May 25 13:10:10.637730 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:10.637681 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" event={"ID":"c0b3f96f-81f4-4063-a627-069319cc24a1","Type":"ContainerStarted","Data":"072c2dcc7334623e8cbb270e70f46d26042e92e48bc5462ea9251408d35e9458"} May 25 13:10:12.645798 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:12.645758 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" event={"ID":"c0b3f96f-81f4-4063-a627-069319cc24a1","Type":"ContainerStarted","Data":"22d4832e296b63ff78fe78aafc1a5e240c2b95007575203cc60d014b8b590321"} May 25 13:10:12.645798 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:12.645798 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" event={"ID":"c0b3f96f-81f4-4063-a627-069319cc24a1","Type":"ContainerStarted","Data":"83dd99f2b44026a39c07c92e20f119211c8d1a3e28644c04bee1201dc9eebcfb"} May 25 13:10:12.646184 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:12.645807 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" event={"ID":"c0b3f96f-81f4-4063-a627-069319cc24a1","Type":"ContainerStarted","Data":"7bdf8e889be67aeb6c0fe5e0a5e57f746ec93d158ec2e12d298186b882ec5969"} May 25 13:10:12.670158 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:12.670114 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75f9b66f-svp5b" podStartSLOduration=1.781733864 podStartE2EDuration="3.670100558s" podCreationTimestamp="2026-05-25 13:10:09 +0000 UTC" firstStartedPulling="2026-05-25 13:10:10.048292882 +0000 UTC m=+221.726590465" lastFinishedPulling="2026-05-25 13:10:11.936659579 +0000 UTC m=+223.614957159" observedRunningTime="2026-05-25 13:10:12.668710166 +0000 UTC m=+224.347007768" watchObservedRunningTime="2026-05-25 13:10:12.670100558 +0000 UTC m=+224.348398159" May 25 13:10:13.324909 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.324871 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6998c7b4dd-s9k4k"] May 25 13:10:13.327554 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.327528 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.338409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.338385 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6998c7b4dd-s9k4k"] May 25 13:10:13.357055 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.357026 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-service-ca\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.357055 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.357057 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-trusted-ca-bundle\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.357208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.357085 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-serving-cert\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.357208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.357104 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-config\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.357208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.357160 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zqj\" (UniqueName: \"kubernetes.io/projected/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-kube-api-access-z6zqj\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.357313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.357208 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-oauth-config\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.357313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.357226 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-oauth-serving-cert\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.457940 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.457913 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-oauth-config\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.457940 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.457943 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-oauth-serving-cert\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.457991 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-service-ca\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458026 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-trusted-ca-bundle\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458069 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-serving-cert\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458197 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-config\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458265 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458236 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zqj\" (UniqueName: \"kubernetes.io/projected/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-kube-api-access-z6zqj\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458731 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458711 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-oauth-serving-cert\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458834 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458807 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-service-ca\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458923 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458905 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-config\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.458990 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.458954 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-trusted-ca-bundle\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.460471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.460444 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-serving-cert\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.460573 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.460471 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-oauth-config\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.466343 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.466322 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zqj\" (UniqueName: \"kubernetes.io/projected/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-kube-api-access-z6zqj\") pod \"console-6998c7b4dd-s9k4k\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.637335 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.637275 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:13.771911 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:13.771882 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6998c7b4dd-s9k4k"] May 25 13:10:13.774568 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:10:13.774543 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac3e9b3_df0a_4882_ad47_3ed7f9d7adac.slice/crio-c72f0de9fe4a29e0a951edd10b61eab46944ec44395f8b084a9220307d1b1e2a WatchSource:0}: Error finding container c72f0de9fe4a29e0a951edd10b61eab46944ec44395f8b084a9220307d1b1e2a: Status 404 returned error can't find the container with id c72f0de9fe4a29e0a951edd10b61eab46944ec44395f8b084a9220307d1b1e2a May 25 13:10:14.653723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:14.653690 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6998c7b4dd-s9k4k" event={"ID":"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac","Type":"ContainerStarted","Data":"e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6"} May 25 13:10:14.653723 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:14.653728 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6998c7b4dd-s9k4k" event={"ID":"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac","Type":"ContainerStarted","Data":"c72f0de9fe4a29e0a951edd10b61eab46944ec44395f8b084a9220307d1b1e2a"} May 25 13:10:14.672002 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:14.671951 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6998c7b4dd-s9k4k" podStartSLOduration=1.671937086 podStartE2EDuration="1.671937086s" podCreationTimestamp="2026-05-25 13:10:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:10:14.671004857 +0000 UTC m=+226.349302461" watchObservedRunningTime="2026-05-25 13:10:14.671937086 +0000 UTC m=+226.350234687" May 25 13:10:23.637799 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:23.637755 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:23.637799 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:23.637804 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:23.642809 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:23.642781 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:23.695592 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:23.695565 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:10:23.737120 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:23.737094 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5489cdd44b-kfmlp"] May 25 13:10:48.756287 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:48.756191 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5489cdd44b-kfmlp" podUID="c38e49d4-c25c-4273-aa8b-c73912c1e43f" containerName="console" containerID="cri-o://355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8" gracePeriod=15 May 25 13:10:49.004012 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.003988 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5489cdd44b-kfmlp_c38e49d4-c25c-4273-aa8b-c73912c1e43f/console/0.log" May 25 13:10:49.004126 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.004049 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:10:49.100285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100258 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-oauth-config\") pod \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " May 25 13:10:49.100441 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100291 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-config\") pod \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " May 25 13:10:49.100441 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100332 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-serving-cert\") pod \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " May 25 13:10:49.100441 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100350 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnbxk\" (UniqueName: \"kubernetes.io/projected/c38e49d4-c25c-4273-aa8b-c73912c1e43f-kube-api-access-wnbxk\") pod \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " May 25 13:10:49.100441 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100391 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-trusted-ca-bundle\") pod \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " May 25 13:10:49.100441 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100419 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-oauth-serving-cert\") pod \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " May 25 13:10:49.100681 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100475 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-service-ca\") pod \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\" (UID: \"c38e49d4-c25c-4273-aa8b-c73912c1e43f\") " May 25 13:10:49.100920 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.100880 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-config" (OuterVolumeSpecName: "console-config") pod "c38e49d4-c25c-4273-aa8b-c73912c1e43f" (UID: "c38e49d4-c25c-4273-aa8b-c73912c1e43f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:10:49.101054 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.101000 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-service-ca" (OuterVolumeSpecName: "service-ca") pod "c38e49d4-c25c-4273-aa8b-c73912c1e43f" (UID: "c38e49d4-c25c-4273-aa8b-c73912c1e43f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:10:49.101054 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.101014 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c38e49d4-c25c-4273-aa8b-c73912c1e43f" (UID: "c38e49d4-c25c-4273-aa8b-c73912c1e43f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:10:49.101054 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.101037 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c38e49d4-c25c-4273-aa8b-c73912c1e43f" (UID: "c38e49d4-c25c-4273-aa8b-c73912c1e43f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:10:49.102570 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.102553 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c38e49d4-c25c-4273-aa8b-c73912c1e43f" (UID: "c38e49d4-c25c-4273-aa8b-c73912c1e43f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:49.102996 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.102972 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38e49d4-c25c-4273-aa8b-c73912c1e43f-kube-api-access-wnbxk" (OuterVolumeSpecName: "kube-api-access-wnbxk") pod "c38e49d4-c25c-4273-aa8b-c73912c1e43f" (UID: "c38e49d4-c25c-4273-aa8b-c73912c1e43f"). InnerVolumeSpecName "kube-api-access-wnbxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:10:49.102996 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.102984 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c38e49d4-c25c-4273-aa8b-c73912c1e43f" (UID: "c38e49d4-c25c-4273-aa8b-c73912c1e43f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:10:49.201924 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.201901 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-oauth-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:49.201924 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.201922 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-service-ca\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:49.202045 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.201932 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-oauth-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:49.202045 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.201941 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:49.202045 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.201949 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38e49d4-c25c-4273-aa8b-c73912c1e43f-console-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:49.202045 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.201958 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbxk\" (UniqueName: \"kubernetes.io/projected/c38e49d4-c25c-4273-aa8b-c73912c1e43f-kube-api-access-wnbxk\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:49.202045 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.201967 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38e49d4-c25c-4273-aa8b-c73912c1e43f-trusted-ca-bundle\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:10:49.779098 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.779072 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5489cdd44b-kfmlp_c38e49d4-c25c-4273-aa8b-c73912c1e43f/console/0.log" May 25 13:10:49.779474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.779110 2554 generic.go:358] "Generic (PLEG): container finished" podID="c38e49d4-c25c-4273-aa8b-c73912c1e43f" containerID="355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8" exitCode=2 May 25 13:10:49.779474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.779176 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5489cdd44b-kfmlp" May 25 13:10:49.779474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.779180 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5489cdd44b-kfmlp" event={"ID":"c38e49d4-c25c-4273-aa8b-c73912c1e43f","Type":"ContainerDied","Data":"355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8"} May 25 13:10:49.779474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.779211 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5489cdd44b-kfmlp" event={"ID":"c38e49d4-c25c-4273-aa8b-c73912c1e43f","Type":"ContainerDied","Data":"346f811cb79beee1e93cd9c9b99d7b60e9ccdd4a9f694f4f033a1ce5e6388820"} May 25 13:10:49.779474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.779230 2554 scope.go:117] "RemoveContainer" containerID="355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8" May 25 13:10:49.787064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.787045 2554 scope.go:117] "RemoveContainer" containerID="355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8" May 25 13:10:49.787282 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:10:49.787264 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8\": container with ID starting with 355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8 not found: ID does not exist" containerID="355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8" May 25 13:10:49.787368 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.787288 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8"} err="failed to get container status \"355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8\": rpc error: code = NotFound desc = could not find container \"355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8\": container with ID starting with 355459ed84e2b600ee8cafc6b306a4ab9972cbdd2506b3634ad8d7424f550fe8 not found: ID does not exist" May 25 13:10:49.799118 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.799098 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5489cdd44b-kfmlp"] May 25 13:10:49.804475 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:49.804458 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5489cdd44b-kfmlp"] May 25 13:10:50.901384 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:10:50.901348 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c38e49d4-c25c-4273-aa8b-c73912c1e43f" path="/var/lib/kubelet/pods/c38e49d4-c25c-4273-aa8b-c73912c1e43f/volumes" May 25 13:11:21.468175 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.468141 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7df544469f-crvl5"] May 25 13:11:21.468625 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.468453 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38e49d4-c25c-4273-aa8b-c73912c1e43f" containerName="console" May 25 13:11:21.468625 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.468465 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38e49d4-c25c-4273-aa8b-c73912c1e43f" containerName="console" May 25 13:11:21.468625 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.468528 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="c38e49d4-c25c-4273-aa8b-c73912c1e43f" containerName="console" May 25 13:11:21.471940 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.471898 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.485994 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.485968 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df544469f-crvl5"] May 25 13:11:21.648325 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.648290 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-console-config\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.648528 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.648329 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-oauth-config\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.648528 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.648362 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-serving-cert\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.648528 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.648462 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-service-ca\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.648528 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.648492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn29z\" (UniqueName: \"kubernetes.io/projected/99aca555-4465-4182-8cb1-da88e32fda75-kube-api-access-pn29z\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.648528 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.648520 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-oauth-serving-cert\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.648744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.648602 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-trusted-ca-bundle\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.749864 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.749809 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-oauth-config\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.749864 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.749840 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-serving-cert\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.749870 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-service-ca\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.749887 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pn29z\" (UniqueName: \"kubernetes.io/projected/99aca555-4465-4182-8cb1-da88e32fda75-kube-api-access-pn29z\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.749909 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-oauth-serving-cert\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.749935 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-trusted-ca-bundle\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.749968 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-console-config\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750721 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.750695 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-service-ca\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750818 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.750695 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-oauth-serving-cert\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750818 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.750762 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-console-config\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.750991 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.750970 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-trusted-ca-bundle\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.752148 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.752121 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-oauth-config\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.752416 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.752398 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-serving-cert\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.759821 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.759801 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn29z\" (UniqueName: \"kubernetes.io/projected/99aca555-4465-4182-8cb1-da88e32fda75-kube-api-access-pn29z\") pod \"console-7df544469f-crvl5\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.782605 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.782583 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:21.910198 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:21.910132 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df544469f-crvl5"] May 25 13:11:21.912810 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:11:21.912783 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99aca555_4465_4182_8cb1_da88e32fda75.slice/crio-e6ec970ea98bedc41c7f9c1eefba101679a5cc89f2af5e4537870f2c52ebbb21 WatchSource:0}: Error finding container e6ec970ea98bedc41c7f9c1eefba101679a5cc89f2af5e4537870f2c52ebbb21: Status 404 returned error can't find the container with id e6ec970ea98bedc41c7f9c1eefba101679a5cc89f2af5e4537870f2c52ebbb21 May 25 13:11:22.884476 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:22.884440 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df544469f-crvl5" event={"ID":"99aca555-4465-4182-8cb1-da88e32fda75","Type":"ContainerStarted","Data":"84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321"} May 25 13:11:22.884476 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:22.884478 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df544469f-crvl5" event={"ID":"99aca555-4465-4182-8cb1-da88e32fda75","Type":"ContainerStarted","Data":"e6ec970ea98bedc41c7f9c1eefba101679a5cc89f2af5e4537870f2c52ebbb21"} May 25 13:11:22.902968 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:22.902925 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7df544469f-crvl5" podStartSLOduration=1.9028935059999998 podStartE2EDuration="1.902893506s" podCreationTimestamp="2026-05-25 13:11:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:11:22.90278864 +0000 UTC m=+294.581086243" watchObservedRunningTime="2026-05-25 13:11:22.902893506 +0000 UTC m=+294.581191109" May 25 13:11:28.788565 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:28.788530 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:11:28.789200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:28.789032 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:11:28.799041 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:28.799017 2554 kubelet.go:1628] "Image garbage collection succeeded" May 25 13:11:31.782841 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:31.782794 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:31.782841 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:31.782841 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:31.787848 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:31.787825 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:31.914109 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:31.914084 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7df544469f-crvl5" May 25 13:11:31.970358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:31.970319 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6998c7b4dd-s9k4k"] May 25 13:11:56.990467 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:56.990399 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6998c7b4dd-s9k4k" podUID="aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" containerName="console" containerID="cri-o://e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6" gracePeriod=15 May 25 13:11:57.234708 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.234687 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6998c7b4dd-s9k4k_aac3e9b3-df0a-4882-ad47-3ed7f9d7adac/console/0.log" May 25 13:11:57.234831 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.234759 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:11:57.301278 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301217 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-config\") pod \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " May 25 13:11:57.301278 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301266 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-trusted-ca-bundle\") pod \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " May 25 13:11:57.301463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301293 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6zqj\" (UniqueName: \"kubernetes.io/projected/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-kube-api-access-z6zqj\") pod \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " May 25 13:11:57.301463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301327 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-serving-cert\") pod \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " May 25 13:11:57.301463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301360 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-service-ca\") pod \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " May 25 13:11:57.301463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301383 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-oauth-config\") pod \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " May 25 13:11:57.301666 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301473 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-oauth-serving-cert\") pod \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\" (UID: \"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac\") " May 25 13:11:57.301719 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301701 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-config" (OuterVolumeSpecName: "console-config") pod "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" (UID: "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:11:57.301775 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301747 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-service-ca" (OuterVolumeSpecName: "service-ca") pod "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" (UID: "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:11:57.301831 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301780 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" (UID: "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:11:57.301998 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.301969 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" (UID: "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:11:57.303457 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.303438 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" (UID: "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:11:57.303523 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.303504 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-kube-api-access-z6zqj" (OuterVolumeSpecName: "kube-api-access-z6zqj") pod "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" (UID: "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac"). InnerVolumeSpecName "kube-api-access-z6zqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:11:57.303608 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.303591 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" (UID: "aac3e9b3-df0a-4882-ad47-3ed7f9d7adac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:11:57.402099 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.402075 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:11:57.402099 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.402097 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-trusted-ca-bundle\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:11:57.402219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.402107 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zqj\" (UniqueName: \"kubernetes.io/projected/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-kube-api-access-z6zqj\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:11:57.402219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.402117 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:11:57.402219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.402126 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-service-ca\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:11:57.402219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.402137 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-console-oauth-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:11:57.402219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.402145 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac-oauth-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:11:57.981811 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.981785 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6998c7b4dd-s9k4k_aac3e9b3-df0a-4882-ad47-3ed7f9d7adac/console/0.log" May 25 13:11:57.981966 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.981825 2554 generic.go:358] "Generic (PLEG): container finished" podID="aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" containerID="e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6" exitCode=2 May 25 13:11:57.981966 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.981892 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6998c7b4dd-s9k4k" May 25 13:11:57.982039 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.981893 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6998c7b4dd-s9k4k" event={"ID":"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac","Type":"ContainerDied","Data":"e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6"} May 25 13:11:57.982039 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.981996 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6998c7b4dd-s9k4k" event={"ID":"aac3e9b3-df0a-4882-ad47-3ed7f9d7adac","Type":"ContainerDied","Data":"c72f0de9fe4a29e0a951edd10b61eab46944ec44395f8b084a9220307d1b1e2a"} May 25 13:11:57.982039 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.982012 2554 scope.go:117] "RemoveContainer" containerID="e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6" May 25 13:11:57.992351 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.992138 2554 scope.go:117] "RemoveContainer" containerID="e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6" May 25 13:11:57.992655 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:11:57.992486 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6\": container with ID starting with e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6 not found: ID does not exist" containerID="e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6" May 25 13:11:57.992655 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:57.992514 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6"} err="failed to get container status \"e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6\": rpc error: code = NotFound desc = could not find container \"e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6\": container with ID starting with e2b27dc4691a37266737a9db3a03c4816cd8d0d6263c8d5147c32f4535c1a3b6 not found: ID does not exist" May 25 13:11:58.003871 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:58.003851 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6998c7b4dd-s9k4k"] May 25 13:11:58.007307 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:58.007286 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6998c7b4dd-s9k4k"] May 25 13:11:58.900477 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:11:58.900446 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" path="/var/lib/kubelet/pods/aac3e9b3-df0a-4882-ad47-3ed7f9d7adac/volumes" May 25 13:12:02.953441 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:02.953397 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-khdhn"] May 25 13:12:02.953861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:02.953692 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" containerName="console" May 25 13:12:02.953861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:02.953704 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" containerName="console" May 25 13:12:02.953861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:02.953755 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="aac3e9b3-df0a-4882-ad47-3ed7f9d7adac" containerName="console" May 25 13:12:02.956570 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:02.956551 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:02.958698 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:02.958676 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 25 13:12:02.971706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:02.971684 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-khdhn"] May 25 13:12:03.041495 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.041472 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db21c73e-1377-495c-b026-629cacfc1a70-kubelet-config\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.041584 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.041517 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db21c73e-1377-495c-b026-629cacfc1a70-original-pull-secret\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.041625 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.041606 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db21c73e-1377-495c-b026-629cacfc1a70-dbus\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.142171 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.142149 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db21c73e-1377-495c-b026-629cacfc1a70-dbus\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.142252 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.142196 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db21c73e-1377-495c-b026-629cacfc1a70-kubelet-config\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.142252 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.142215 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db21c73e-1377-495c-b026-629cacfc1a70-original-pull-secret\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.142331 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.142271 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db21c73e-1377-495c-b026-629cacfc1a70-kubelet-config\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.142368 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.142337 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db21c73e-1377-495c-b026-629cacfc1a70-dbus\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.144325 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.144308 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db21c73e-1377-495c-b026-629cacfc1a70-original-pull-secret\") pod \"global-pull-secret-syncer-khdhn\" (UID: \"db21c73e-1377-495c-b026-629cacfc1a70\") " pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.265909 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.265837 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-khdhn" May 25 13:12:03.393054 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.393027 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-khdhn"] May 25 13:12:03.395994 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:12:03.395971 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb21c73e_1377_495c_b026_629cacfc1a70.slice/crio-b8843a2c9894420b781299ee51d9040f490e1f4605b7ec83444677858d9e9ec7 WatchSource:0}: Error finding container b8843a2c9894420b781299ee51d9040f490e1f4605b7ec83444677858d9e9ec7: Status 404 returned error can't find the container with id b8843a2c9894420b781299ee51d9040f490e1f4605b7ec83444677858d9e9ec7 May 25 13:12:03.397466 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:03.397452 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 13:12:04.000753 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:04.000714 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-khdhn" event={"ID":"db21c73e-1377-495c-b026-629cacfc1a70","Type":"ContainerStarted","Data":"b8843a2c9894420b781299ee51d9040f490e1f4605b7ec83444677858d9e9ec7"} May 25 13:12:08.014873 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:08.014833 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-khdhn" event={"ID":"db21c73e-1377-495c-b026-629cacfc1a70","Type":"ContainerStarted","Data":"c4c4f25e9ed9889b9e82f9de0c42528218d025435179f3a172da71b742dfb285"} May 25 13:12:08.034516 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:12:08.034463 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-khdhn" podStartSLOduration=2.193421928 podStartE2EDuration="6.034445909s" podCreationTimestamp="2026-05-25 13:12:02 +0000 UTC" firstStartedPulling="2026-05-25 13:12:03.397573231 +0000 UTC m=+335.075870811" lastFinishedPulling="2026-05-25 13:12:07.238597208 +0000 UTC m=+338.916894792" observedRunningTime="2026-05-25 13:12:08.034048923 +0000 UTC m=+339.712346527" watchObservedRunningTime="2026-05-25 13:12:08.034445909 +0000 UTC m=+339.712743504" May 25 13:15:38.851864 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.851827 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-74bb8fbbdc-vk76q"] May 25 13:15:38.855585 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.855553 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:38.888885 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.888857 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bb8fbbdc-vk76q"] May 25 13:15:38.989654 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.989622 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-oauth-serving-cert\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:38.989808 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.989689 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3997723e-5f8f-4631-b047-89eeb7b79978-console-oauth-config\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:38.989808 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.989705 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-service-ca\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:38.989808 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.989721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3997723e-5f8f-4631-b047-89eeb7b79978-console-serving-cert\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:38.989808 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.989737 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-console-config\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:38.989808 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.989767 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-trusted-ca-bundle\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:38.990073 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:38.989833 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtjv\" (UniqueName: \"kubernetes.io/projected/3997723e-5f8f-4631-b047-89eeb7b79978-kube-api-access-8wtjv\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.090358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.090324 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3997723e-5f8f-4631-b047-89eeb7b79978-console-oauth-config\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.090515 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.090365 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-service-ca\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.090515 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.090393 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3997723e-5f8f-4631-b047-89eeb7b79978-console-serving-cert\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.090515 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.090419 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-console-config\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.090515 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.090495 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-trusted-ca-bundle\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.090720 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.090531 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtjv\" (UniqueName: \"kubernetes.io/projected/3997723e-5f8f-4631-b047-89eeb7b79978-kube-api-access-8wtjv\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.090720 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.090583 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-oauth-serving-cert\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.091179 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.091143 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-service-ca\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.091285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.091224 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-oauth-serving-cert\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.091285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.091261 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-console-config\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.091619 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.091601 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3997723e-5f8f-4631-b047-89eeb7b79978-trusted-ca-bundle\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.093296 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.093274 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3997723e-5f8f-4631-b047-89eeb7b79978-console-serving-cert\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.093399 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.093381 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3997723e-5f8f-4631-b047-89eeb7b79978-console-oauth-config\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.099864 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.099840 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtjv\" (UniqueName: \"kubernetes.io/projected/3997723e-5f8f-4631-b047-89eeb7b79978-kube-api-access-8wtjv\") pod \"console-74bb8fbbdc-vk76q\" (UID: \"3997723e-5f8f-4631-b047-89eeb7b79978\") " pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.164795 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.164750 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:39.294790 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.294762 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bb8fbbdc-vk76q"] May 25 13:15:39.298610 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:15:39.298584 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3997723e_5f8f_4631_b047_89eeb7b79978.slice/crio-8d71156990f202b513f404079cb139200b227ad935fe51c18b7748c37a353b87 WatchSource:0}: Error finding container 8d71156990f202b513f404079cb139200b227ad935fe51c18b7748c37a353b87: Status 404 returned error can't find the container with id 8d71156990f202b513f404079cb139200b227ad935fe51c18b7748c37a353b87 May 25 13:15:39.658049 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.658015 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bb8fbbdc-vk76q" event={"ID":"3997723e-5f8f-4631-b047-89eeb7b79978","Type":"ContainerStarted","Data":"e5537495632521d84ca3b2fbd9687ce18f1406ae1ef718ea8f67c9ecb2f5f824"} May 25 13:15:39.658049 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.658052 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bb8fbbdc-vk76q" event={"ID":"3997723e-5f8f-4631-b047-89eeb7b79978","Type":"ContainerStarted","Data":"8d71156990f202b513f404079cb139200b227ad935fe51c18b7748c37a353b87"} May 25 13:15:39.677213 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:39.677170 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-74bb8fbbdc-vk76q" podStartSLOduration=1.677157034 podStartE2EDuration="1.677157034s" podCreationTimestamp="2026-05-25 13:15:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:15:39.676022212 +0000 UTC m=+551.354319816" watchObservedRunningTime="2026-05-25 13:15:39.677157034 +0000 UTC m=+551.355454636" May 25 13:15:49.165354 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:49.165321 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:49.165354 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:49.165362 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:49.169957 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:49.169932 2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:49.696124 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:49.696095 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74bb8fbbdc-vk76q" May 25 13:15:49.758313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:15:49.758279 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df544469f-crvl5"] May 25 13:16:14.777283 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:14.777227 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7df544469f-crvl5" podUID="99aca555-4465-4182-8cb1-da88e32fda75" containerName="console" containerID="cri-o://84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321" gracePeriod=15 May 25 13:16:15.010495 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.010471 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7df544469f-crvl5_99aca555-4465-4182-8cb1-da88e32fda75/console/0.log" May 25 13:16:15.010601 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.010531 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df544469f-crvl5" May 25 13:16:15.160526 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160501 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-console-config\") pod \"99aca555-4465-4182-8cb1-da88e32fda75\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " May 25 13:16:15.160646 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160530 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-serving-cert\") pod \"99aca555-4465-4182-8cb1-da88e32fda75\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " May 25 13:16:15.160646 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160549 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-oauth-serving-cert\") pod \"99aca555-4465-4182-8cb1-da88e32fda75\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " May 25 13:16:15.160646 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160579 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn29z\" (UniqueName: \"kubernetes.io/projected/99aca555-4465-4182-8cb1-da88e32fda75-kube-api-access-pn29z\") pod \"99aca555-4465-4182-8cb1-da88e32fda75\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " May 25 13:16:15.160750 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160691 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-trusted-ca-bundle\") pod \"99aca555-4465-4182-8cb1-da88e32fda75\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " May 25 13:16:15.160750 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160731 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-service-ca\") pod \"99aca555-4465-4182-8cb1-da88e32fda75\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " May 25 13:16:15.160853 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160760 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-oauth-config\") pod \"99aca555-4465-4182-8cb1-da88e32fda75\" (UID: \"99aca555-4465-4182-8cb1-da88e32fda75\") " May 25 13:16:15.161026 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.160995 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-console-config" (OuterVolumeSpecName: "console-config") pod "99aca555-4465-4182-8cb1-da88e32fda75" (UID: "99aca555-4465-4182-8cb1-da88e32fda75"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:16:15.161026 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.161005 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "99aca555-4465-4182-8cb1-da88e32fda75" (UID: "99aca555-4465-4182-8cb1-da88e32fda75"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:16:15.161174 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.161115 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-service-ca" (OuterVolumeSpecName: "service-ca") pod "99aca555-4465-4182-8cb1-da88e32fda75" (UID: "99aca555-4465-4182-8cb1-da88e32fda75"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:16:15.161219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.161176 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "99aca555-4465-4182-8cb1-da88e32fda75" (UID: "99aca555-4465-4182-8cb1-da88e32fda75"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:16:15.162984 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.162964 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "99aca555-4465-4182-8cb1-da88e32fda75" (UID: "99aca555-4465-4182-8cb1-da88e32fda75"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:16:15.163176 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.163157 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "99aca555-4465-4182-8cb1-da88e32fda75" (UID: "99aca555-4465-4182-8cb1-da88e32fda75"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:16:15.163176 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.163163 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99aca555-4465-4182-8cb1-da88e32fda75-kube-api-access-pn29z" (OuterVolumeSpecName: "kube-api-access-pn29z") pod "99aca555-4465-4182-8cb1-da88e32fda75" (UID: "99aca555-4465-4182-8cb1-da88e32fda75"). InnerVolumeSpecName "kube-api-access-pn29z". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:16:15.262189 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.262166 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-service-ca\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:16:15.262189 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.262188 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-oauth-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:16:15.262313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.262197 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-console-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:16:15.262313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.262206 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99aca555-4465-4182-8cb1-da88e32fda75-console-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:16:15.262313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.262214 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-oauth-serving-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:16:15.262313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.262227 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn29z\" (UniqueName: \"kubernetes.io/projected/99aca555-4465-4182-8cb1-da88e32fda75-kube-api-access-pn29z\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:16:15.262313 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.262240 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99aca555-4465-4182-8cb1-da88e32fda75-trusted-ca-bundle\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:16:15.779388 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.779365 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7df544469f-crvl5_99aca555-4465-4182-8cb1-da88e32fda75/console/0.log" May 25 13:16:15.779751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.779401 2554 generic.go:358] "Generic (PLEG): container finished" podID="99aca555-4465-4182-8cb1-da88e32fda75" containerID="84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321" exitCode=2 May 25 13:16:15.779751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.779484 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df544469f-crvl5" event={"ID":"99aca555-4465-4182-8cb1-da88e32fda75","Type":"ContainerDied","Data":"84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321"} May 25 13:16:15.779751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.779511 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df544469f-crvl5" event={"ID":"99aca555-4465-4182-8cb1-da88e32fda75","Type":"ContainerDied","Data":"e6ec970ea98bedc41c7f9c1eefba101679a5cc89f2af5e4537870f2c52ebbb21"} May 25 13:16:15.779751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.779525 2554 scope.go:117] "RemoveContainer" containerID="84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321" May 25 13:16:15.779751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.779486 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df544469f-crvl5" May 25 13:16:15.788474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.788453 2554 scope.go:117] "RemoveContainer" containerID="84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321" May 25 13:16:15.788701 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:16:15.788683 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321\": container with ID starting with 84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321 not found: ID does not exist" containerID="84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321" May 25 13:16:15.788749 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.788709 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321"} err="failed to get container status \"84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321\": rpc error: code = NotFound desc = could not find container \"84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321\": container with ID starting with 84b41a2014e95fd2475dea57b62c73f2c20a9b621851fd799e076763b2da2321 not found: ID does not exist" May 25 13:16:15.800341 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.800316 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df544469f-crvl5"] May 25 13:16:15.806050 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:15.806029 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7df544469f-crvl5"] May 25 13:16:16.900439 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:16.900400 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99aca555-4465-4182-8cb1-da88e32fda75" path="/var/lib/kubelet/pods/99aca555-4465-4182-8cb1-da88e32fda75/volumes" May 25 13:16:28.812912 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:28.812882 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:16:28.814405 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:28.814372 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:16:53.276993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.276910 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2"] May 25 13:16:53.279369 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.277267 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99aca555-4465-4182-8cb1-da88e32fda75" containerName="console" May 25 13:16:53.279369 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.277280 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="99aca555-4465-4182-8cb1-da88e32fda75" containerName="console" May 25 13:16:53.279369 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.277355 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="99aca555-4465-4182-8cb1-da88e32fda75" containerName="console" May 25 13:16:53.280382 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.280364 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.282778 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.282756 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drkvx\"" May 25 13:16:53.283051 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.283034 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" May 25 13:16:53.283595 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.283574 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" May 25 13:16:53.293601 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.293578 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2"] May 25 13:16:53.327172 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.327152 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.327272 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.327184 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.327272 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.327203 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q8lk\" (UniqueName: \"kubernetes.io/projected/7893cb9c-a7f9-4920-aee4-3c361629b49e-kube-api-access-9q8lk\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.428401 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.428379 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.428523 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.428414 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.428523 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.428456 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9q8lk\" (UniqueName: \"kubernetes.io/projected/7893cb9c-a7f9-4920-aee4-3c361629b49e-kube-api-access-9q8lk\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.428722 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.428703 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.428802 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.428782 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.437788 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.437766 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q8lk\" (UniqueName: \"kubernetes.io/projected/7893cb9c-a7f9-4920-aee4-3c361629b49e-kube-api-access-9q8lk\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.590006 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.589978 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:16:53.711260 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.711229 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2"] May 25 13:16:53.714467 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:16:53.714438 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7893cb9c_a7f9_4920_aee4_3c361629b49e.slice/crio-ecbdc272121b2a8e99f01c9b787733c5d6490149a891d165f0aa626620bfbabe WatchSource:0}: Error finding container ecbdc272121b2a8e99f01c9b787733c5d6490149a891d165f0aa626620bfbabe: Status 404 returned error can't find the container with id ecbdc272121b2a8e99f01c9b787733c5d6490149a891d165f0aa626620bfbabe May 25 13:16:53.901073 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:53.901010 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" event={"ID":"7893cb9c-a7f9-4920-aee4-3c361629b49e","Type":"ContainerStarted","Data":"ecbdc272121b2a8e99f01c9b787733c5d6490149a891d165f0aa626620bfbabe"} May 25 13:16:59.920910 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:59.920871 2554 generic.go:358] "Generic (PLEG): container finished" podID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerID="edf3722b9673a16df0cc6a8a2014e5df8b30ceee78e1eda84db2b0575a2483ef" exitCode=0 May 25 13:16:59.921268 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:16:59.920959 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" event={"ID":"7893cb9c-a7f9-4920-aee4-3c361629b49e","Type":"ContainerDied","Data":"edf3722b9673a16df0cc6a8a2014e5df8b30ceee78e1eda84db2b0575a2483ef"} May 25 13:17:02.929868 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:02.929833 2554 generic.go:358] "Generic (PLEG): container finished" podID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerID="2acb4557fcfbc58d2c3cf079737c6a4596e90a7a91ff9d60c29cd07c3465cc7e" exitCode=0 May 25 13:17:02.930255 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:02.929914 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" event={"ID":"7893cb9c-a7f9-4920-aee4-3c361629b49e","Type":"ContainerDied","Data":"2acb4557fcfbc58d2c3cf079737c6a4596e90a7a91ff9d60c29cd07c3465cc7e"} May 25 13:17:09.955397 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:09.955363 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" event={"ID":"7893cb9c-a7f9-4920-aee4-3c361629b49e","Type":"ContainerStarted","Data":"29fcf8272594ea06ac3226884cfc5f371f23ed552777393fc024a73c2ed5e88c"} May 25 13:17:09.975004 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:09.974961 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" podStartSLOduration=0.820263283 podStartE2EDuration="16.97494644s" podCreationTimestamp="2026-05-25 13:16:53 +0000 UTC" firstStartedPulling="2026-05-25 13:16:53.716232747 +0000 UTC m=+625.394530331" lastFinishedPulling="2026-05-25 13:17:09.870915895 +0000 UTC m=+641.549213488" observedRunningTime="2026-05-25 13:17:09.973803821 +0000 UTC m=+641.652101426" watchObservedRunningTime="2026-05-25 13:17:09.97494644 +0000 UTC m=+641.653244060" May 25 13:17:10.960224 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:10.960189 2554 generic.go:358] "Generic (PLEG): container finished" podID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerID="29fcf8272594ea06ac3226884cfc5f371f23ed552777393fc024a73c2ed5e88c" exitCode=0 May 25 13:17:10.960643 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:10.960285 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" event={"ID":"7893cb9c-a7f9-4920-aee4-3c361629b49e","Type":"ContainerDied","Data":"29fcf8272594ea06ac3226884cfc5f371f23ed552777393fc024a73c2ed5e88c"} May 25 13:17:12.086490 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.086465 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:17:12.183023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.182996 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-util\") pod \"7893cb9c-a7f9-4920-aee4-3c361629b49e\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " May 25 13:17:12.183124 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.183039 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q8lk\" (UniqueName: \"kubernetes.io/projected/7893cb9c-a7f9-4920-aee4-3c361629b49e-kube-api-access-9q8lk\") pod \"7893cb9c-a7f9-4920-aee4-3c361629b49e\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " May 25 13:17:12.183124 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.183081 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-bundle\") pod \"7893cb9c-a7f9-4920-aee4-3c361629b49e\" (UID: \"7893cb9c-a7f9-4920-aee4-3c361629b49e\") " May 25 13:17:12.183697 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.183668 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-bundle" (OuterVolumeSpecName: "bundle") pod "7893cb9c-a7f9-4920-aee4-3c361629b49e" (UID: "7893cb9c-a7f9-4920-aee4-3c361629b49e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:17:12.185146 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.185124 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7893cb9c-a7f9-4920-aee4-3c361629b49e-kube-api-access-9q8lk" (OuterVolumeSpecName: "kube-api-access-9q8lk") pod "7893cb9c-a7f9-4920-aee4-3c361629b49e" (UID: "7893cb9c-a7f9-4920-aee4-3c361629b49e"). InnerVolumeSpecName "kube-api-access-9q8lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:17:12.187458 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.187418 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-util" (OuterVolumeSpecName: "util") pod "7893cb9c-a7f9-4920-aee4-3c361629b49e" (UID: "7893cb9c-a7f9-4920-aee4-3c361629b49e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:17:12.284068 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.284020 2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-bundle\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:17:12.284068 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.284041 2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7893cb9c-a7f9-4920-aee4-3c361629b49e-util\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:17:12.284068 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.284050 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q8lk\" (UniqueName: \"kubernetes.io/projected/7893cb9c-a7f9-4920-aee4-3c361629b49e-kube-api-access-9q8lk\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:17:12.967075 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.967049 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" May 25 13:17:12.967075 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.967063 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c4cjn2" event={"ID":"7893cb9c-a7f9-4920-aee4-3c361629b49e","Type":"ContainerDied","Data":"ecbdc272121b2a8e99f01c9b787733c5d6490149a891d165f0aa626620bfbabe"} May 25 13:17:12.967266 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:12.967092 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecbdc272121b2a8e99f01c9b787733c5d6490149a891d165f0aa626620bfbabe" May 25 13:17:15.305053 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305017 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9"] May 25 13:17:15.305549 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305514 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerName="extract" May 25 13:17:15.305549 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305533 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerName="extract" May 25 13:17:15.305549 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305547 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerName="pull" May 25 13:17:15.305714 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305555 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerName="pull" May 25 13:17:15.305714 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305577 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerName="util" May 25 13:17:15.305714 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305586 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerName="util" May 25 13:17:15.305714 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.305669 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="7893cb9c-a7f9-4920-aee4-3c361629b49e" containerName="extract" May 25 13:17:15.310463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.310441 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.314635 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.314612 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" May 25 13:17:15.314731 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.314615 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" May 25 13:17:15.318414 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.318397 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" May 25 13:17:15.319937 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.319914 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-jrpqv\"" May 25 13:17:15.326876 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.326857 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9"] May 25 13:17:15.404984 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.404963 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxsdd\" (UniqueName: \"kubernetes.io/projected/75e55101-6355-4224-ba6b-0fdbb092b85e-kube-api-access-qxsdd\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-29vh9\" (UID: \"75e55101-6355-4224-ba6b-0fdbb092b85e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.405162 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.404997 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/75e55101-6355-4224-ba6b-0fdbb092b85e-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-29vh9\" (UID: \"75e55101-6355-4224-ba6b-0fdbb092b85e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.505439 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.505408 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qxsdd\" (UniqueName: \"kubernetes.io/projected/75e55101-6355-4224-ba6b-0fdbb092b85e-kube-api-access-qxsdd\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-29vh9\" (UID: \"75e55101-6355-4224-ba6b-0fdbb092b85e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.505543 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.505475 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/75e55101-6355-4224-ba6b-0fdbb092b85e-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-29vh9\" (UID: \"75e55101-6355-4224-ba6b-0fdbb092b85e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.507674 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.507654 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/75e55101-6355-4224-ba6b-0fdbb092b85e-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-29vh9\" (UID: \"75e55101-6355-4224-ba6b-0fdbb092b85e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.515260 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.515241 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxsdd\" (UniqueName: \"kubernetes.io/projected/75e55101-6355-4224-ba6b-0fdbb092b85e-kube-api-access-qxsdd\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-29vh9\" (UID: \"75e55101-6355-4224-ba6b-0fdbb092b85e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.620935 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.620894 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:15.755048 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.755020 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9"] May 25 13:17:15.757533 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:17:15.757497 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e55101_6355_4224_ba6b_0fdbb092b85e.slice/crio-9aa1281fddc4f641c914ea626f990e4ac2b63c659180f6fefbeb8a3682615b62 WatchSource:0}: Error finding container 9aa1281fddc4f641c914ea626f990e4ac2b63c659180f6fefbeb8a3682615b62: Status 404 returned error can't find the container with id 9aa1281fddc4f641c914ea626f990e4ac2b63c659180f6fefbeb8a3682615b62 May 25 13:17:15.759236 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.759217 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 13:17:15.977342 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:15.977279 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" event={"ID":"75e55101-6355-4224-ba6b-0fdbb092b85e","Type":"ContainerStarted","Data":"9aa1281fddc4f641c914ea626f990e4ac2b63c659180f6fefbeb8a3682615b62"} May 25 13:17:19.992756 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:19.992672 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" event={"ID":"75e55101-6355-4224-ba6b-0fdbb092b85e","Type":"ContainerStarted","Data":"22d1c67c7a1735451c6ecb169694059c9cffff8207a85950b2dc008f750d8309"} May 25 13:17:19.993111 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:19.992889 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:20.013208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:20.013166 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" podStartSLOduration=1.053226249 podStartE2EDuration="5.013154392s" podCreationTimestamp="2026-05-25 13:17:15 +0000 UTC" firstStartedPulling="2026-05-25 13:17:15.759401355 +0000 UTC m=+647.437698935" lastFinishedPulling="2026-05-25 13:17:19.719329484 +0000 UTC m=+651.397627078" observedRunningTime="2026-05-25 13:17:20.012639148 +0000 UTC m=+651.690936763" watchObservedRunningTime="2026-05-25 13:17:20.013154392 +0000 UTC m=+651.691451995" May 25 13:17:20.894641 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:20.894600 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-zfxfl"] May 25 13:17:20.897368 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:20.897341 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:20.900535 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:20.900515 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" May 25 13:17:20.900662 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:20.900582 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-2t6ms\"" May 25 13:17:20.901276 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:20.901260 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" May 25 13:17:20.909942 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:20.909912 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-zfxfl"] May 25 13:17:21.048858 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.048831 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwqm\" (UniqueName: \"kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-kube-api-access-5cwqm\") pod \"keda-admission-cf49989db-zfxfl\" (UID: \"7d5b388c-5a67-41ef-b4f8-4d11a529addb\") " pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.049212 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.048895 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-certificates\") pod \"keda-admission-cf49989db-zfxfl\" (UID: \"7d5b388c-5a67-41ef-b4f8-4d11a529addb\") " pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.150297 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.150226 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-certificates\") pod \"keda-admission-cf49989db-zfxfl\" (UID: \"7d5b388c-5a67-41ef-b4f8-4d11a529addb\") " pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.150297 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.150279 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwqm\" (UniqueName: \"kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-kube-api-access-5cwqm\") pod \"keda-admission-cf49989db-zfxfl\" (UID: \"7d5b388c-5a67-41ef-b4f8-4d11a529addb\") " pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.150545 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:17:21.150369 2554 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found May 25 13:17:21.150545 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:17:21.150392 2554 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-zfxfl: secret "keda-admission-webhooks-certs" not found May 25 13:17:21.150545 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:17:21.150461 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-certificates podName:7d5b388c-5a67-41ef-b4f8-4d11a529addb nodeName:}" failed. No retries permitted until 2026-05-25 13:17:21.650444831 +0000 UTC m=+653.328742411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-certificates") pod "keda-admission-cf49989db-zfxfl" (UID: "7d5b388c-5a67-41ef-b4f8-4d11a529addb") : secret "keda-admission-webhooks-certs" not found May 25 13:17:21.160171 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.160149 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwqm\" (UniqueName: \"kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-kube-api-access-5cwqm\") pod \"keda-admission-cf49989db-zfxfl\" (UID: \"7d5b388c-5a67-41ef-b4f8-4d11a529addb\") " pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.654442 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.654395 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-certificates\") pod \"keda-admission-cf49989db-zfxfl\" (UID: \"7d5b388c-5a67-41ef-b4f8-4d11a529addb\") " pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.656890 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.656868 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d5b388c-5a67-41ef-b4f8-4d11a529addb-certificates\") pod \"keda-admission-cf49989db-zfxfl\" (UID: \"7d5b388c-5a67-41ef-b4f8-4d11a529addb\") " pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.808242 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.808209 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:21.939644 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:21.939617 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-zfxfl"] May 25 13:17:21.942349 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:17:21.942321 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5b388c_5a67_41ef_b4f8_4d11a529addb.slice/crio-2968ece592da879a9a309a8a1ee2c47236657178d9387a6af8b4303e5156867e WatchSource:0}: Error finding container 2968ece592da879a9a309a8a1ee2c47236657178d9387a6af8b4303e5156867e: Status 404 returned error can't find the container with id 2968ece592da879a9a309a8a1ee2c47236657178d9387a6af8b4303e5156867e May 25 13:17:22.001038 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:22.001011 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-zfxfl" event={"ID":"7d5b388c-5a67-41ef-b4f8-4d11a529addb","Type":"ContainerStarted","Data":"2968ece592da879a9a309a8a1ee2c47236657178d9387a6af8b4303e5156867e"} May 25 13:17:24.009734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:24.009696 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-zfxfl" event={"ID":"7d5b388c-5a67-41ef-b4f8-4d11a529addb","Type":"ContainerStarted","Data":"ad982893b17b78a92b3d6c8d65d744c7b5d5f8f148ad65da48d6641da2dc5a1d"} May 25 13:17:24.010176 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:24.009811 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:17:24.030211 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:24.030168 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-zfxfl" podStartSLOduration=2.6367705729999997 podStartE2EDuration="4.030155073s" podCreationTimestamp="2026-05-25 13:17:20 +0000 UTC" firstStartedPulling="2026-05-25 13:17:21.943486927 +0000 UTC m=+653.621784507" lastFinishedPulling="2026-05-25 13:17:23.336871424 +0000 UTC m=+655.015169007" observedRunningTime="2026-05-25 13:17:24.029175434 +0000 UTC m=+655.707473033" watchObservedRunningTime="2026-05-25 13:17:24.030155073 +0000 UTC m=+655.708452678" May 25 13:17:40.999409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:40.999371 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-29vh9" May 25 13:17:45.015667 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:17:45.015639 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-zfxfl" May 25 13:18:29.141851 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.141819 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-vb877"] May 25 13:18:29.145019 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.145001 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.147407 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.147384 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" May 25 13:18:29.147900 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.147878 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" May 25 13:18:29.148286 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.148267 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-blms7\"" May 25 13:18:29.148387 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.148300 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" May 25 13:18:29.155491 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.155469 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-vb877"] May 25 13:18:29.254486 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.254460 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31d2a561-feb3-44cb-a2b7-6f4f69008f75-cert\") pod \"llmisvc-controller-manager-6b94ff949c-vb877\" (UID: \"31d2a561-feb3-44cb-a2b7-6f4f69008f75\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.254582 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.254521 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dd72\" (UniqueName: \"kubernetes.io/projected/31d2a561-feb3-44cb-a2b7-6f4f69008f75-kube-api-access-2dd72\") pod \"llmisvc-controller-manager-6b94ff949c-vb877\" (UID: \"31d2a561-feb3-44cb-a2b7-6f4f69008f75\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.355184 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.355154 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2dd72\" (UniqueName: \"kubernetes.io/projected/31d2a561-feb3-44cb-a2b7-6f4f69008f75-kube-api-access-2dd72\") pod \"llmisvc-controller-manager-6b94ff949c-vb877\" (UID: \"31d2a561-feb3-44cb-a2b7-6f4f69008f75\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.355293 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.355238 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31d2a561-feb3-44cb-a2b7-6f4f69008f75-cert\") pod \"llmisvc-controller-manager-6b94ff949c-vb877\" (UID: \"31d2a561-feb3-44cb-a2b7-6f4f69008f75\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.357745 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.357720 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31d2a561-feb3-44cb-a2b7-6f4f69008f75-cert\") pod \"llmisvc-controller-manager-6b94ff949c-vb877\" (UID: \"31d2a561-feb3-44cb-a2b7-6f4f69008f75\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.362821 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.362801 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dd72\" (UniqueName: \"kubernetes.io/projected/31d2a561-feb3-44cb-a2b7-6f4f69008f75-kube-api-access-2dd72\") pod \"llmisvc-controller-manager-6b94ff949c-vb877\" (UID: \"31d2a561-feb3-44cb-a2b7-6f4f69008f75\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.455860 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.455807 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:29.584738 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:29.584703 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-vb877"] May 25 13:18:29.587616 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:18:29.587591 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31d2a561_feb3_44cb_a2b7_6f4f69008f75.slice/crio-c75e1fb78f0d4e9a6bbeae0bb87bd1894b4cde41079d41273971b6e92c2620a2 WatchSource:0}: Error finding container c75e1fb78f0d4e9a6bbeae0bb87bd1894b4cde41079d41273971b6e92c2620a2: Status 404 returned error can't find the container with id c75e1fb78f0d4e9a6bbeae0bb87bd1894b4cde41079d41273971b6e92c2620a2 May 25 13:18:30.234880 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:30.234838 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" event={"ID":"31d2a561-feb3-44cb-a2b7-6f4f69008f75","Type":"ContainerStarted","Data":"c75e1fb78f0d4e9a6bbeae0bb87bd1894b4cde41079d41273971b6e92c2620a2"} May 25 13:18:33.247594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:33.247557 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" event={"ID":"31d2a561-feb3-44cb-a2b7-6f4f69008f75","Type":"ContainerStarted","Data":"57225a5dd483ba046361fabb5d6e220f073b209727c261b63b8de9761b51d14b"} May 25 13:18:33.247976 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:33.247713 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:18:33.266945 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:18:33.266898 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" podStartSLOduration=1.587802336 podStartE2EDuration="4.26688432s" podCreationTimestamp="2026-05-25 13:18:29 +0000 UTC" firstStartedPulling="2026-05-25 13:18:29.589273637 +0000 UTC m=+721.267571220" lastFinishedPulling="2026-05-25 13:18:32.268355609 +0000 UTC m=+723.946653204" observedRunningTime="2026-05-25 13:18:33.264567221 +0000 UTC m=+724.942864823" watchObservedRunningTime="2026-05-25 13:18:33.26688432 +0000 UTC m=+724.945181958" May 25 13:19:04.253521 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:04.253487 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-vb877" May 25 13:19:39.135630 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.135556 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-ml4bh"] May 25 13:19:39.138696 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.138678 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.140990 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.140973 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" May 25 13:19:39.141138 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.141121 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jzq6q\"" May 25 13:19:39.151745 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.151708 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-ml4bh"] May 25 13:19:39.251486 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.251459 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qm55\" (UniqueName: \"kubernetes.io/projected/a4b52f83-ae35-4469-b32c-58a0630496cc-kube-api-access-7qm55\") pod \"odh-model-controller-696fc77849-ml4bh\" (UID: \"a4b52f83-ae35-4469-b32c-58a0630496cc\") " pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.251587 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.251507 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4b52f83-ae35-4469-b32c-58a0630496cc-cert\") pod \"odh-model-controller-696fc77849-ml4bh\" (UID: \"a4b52f83-ae35-4469-b32c-58a0630496cc\") " pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.352270 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.352247 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7qm55\" (UniqueName: \"kubernetes.io/projected/a4b52f83-ae35-4469-b32c-58a0630496cc-kube-api-access-7qm55\") pod \"odh-model-controller-696fc77849-ml4bh\" (UID: \"a4b52f83-ae35-4469-b32c-58a0630496cc\") " pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.352360 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.352285 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4b52f83-ae35-4469-b32c-58a0630496cc-cert\") pod \"odh-model-controller-696fc77849-ml4bh\" (UID: \"a4b52f83-ae35-4469-b32c-58a0630496cc\") " pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.354594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.354576 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4b52f83-ae35-4469-b32c-58a0630496cc-cert\") pod \"odh-model-controller-696fc77849-ml4bh\" (UID: \"a4b52f83-ae35-4469-b32c-58a0630496cc\") " pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.360886 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.360862 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qm55\" (UniqueName: \"kubernetes.io/projected/a4b52f83-ae35-4469-b32c-58a0630496cc-kube-api-access-7qm55\") pod \"odh-model-controller-696fc77849-ml4bh\" (UID: \"a4b52f83-ae35-4469-b32c-58a0630496cc\") " pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.450233 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.450182 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:39.575847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:39.575818 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-ml4bh"] May 25 13:19:39.578500 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:19:39.578477 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b52f83_ae35_4469_b32c_58a0630496cc.slice/crio-db46ff02dd5656ec3ca980c2a67fe16f612f55b4178f67b4d435365dca10b134 WatchSource:0}: Error finding container db46ff02dd5656ec3ca980c2a67fe16f612f55b4178f67b4d435365dca10b134: Status 404 returned error can't find the container with id db46ff02dd5656ec3ca980c2a67fe16f612f55b4178f67b4d435365dca10b134 May 25 13:19:40.490767 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:40.490729 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-ml4bh" event={"ID":"a4b52f83-ae35-4469-b32c-58a0630496cc","Type":"ContainerStarted","Data":"db46ff02dd5656ec3ca980c2a67fe16f612f55b4178f67b4d435365dca10b134"} May 25 13:19:43.503656 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:43.503622 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-ml4bh" event={"ID":"a4b52f83-ae35-4469-b32c-58a0630496cc","Type":"ContainerStarted","Data":"7a6723ce2547f2f386d8f964ee06d67b82e8515d506c332b0d09949ae137ab8b"} May 25 13:19:43.504098 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:43.503725 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:19:43.521103 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:43.521052 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-ml4bh" podStartSLOduration=1.564307597 podStartE2EDuration="4.521041239s" podCreationTimestamp="2026-05-25 13:19:39 +0000 UTC" firstStartedPulling="2026-05-25 13:19:39.579660769 +0000 UTC m=+791.257958353" lastFinishedPulling="2026-05-25 13:19:42.536394412 +0000 UTC m=+794.214691995" observedRunningTime="2026-05-25 13:19:43.519862576 +0000 UTC m=+795.198160181" watchObservedRunningTime="2026-05-25 13:19:43.521041239 +0000 UTC m=+795.199338841" May 25 13:19:54.508805 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:19:54.508773 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-ml4bh" May 25 13:20:16.163733 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.163698 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp"] May 25 13:20:16.167090 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.167074 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.170274 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.170251 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\"" May 25 13:20:16.170594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.170573 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 25 13:20:16.170594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.170584 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 25 13:20:16.170594 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.170580 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-13cfe-predictor-serving-cert\"" May 25 13:20:16.170826 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.170680 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmtw7\"" May 25 13:20:16.182967 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.182943 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp"] May 25 13:20:16.237286 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.237257 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmn4\" (UniqueName: \"kubernetes.io/projected/28622389-8eda-400c-aca2-44177e5f1b71-kube-api-access-fpmn4\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.237396 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.237370 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28622389-8eda-400c-aca2-44177e5f1b71-isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.237471 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.237411 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28622389-8eda-400c-aca2-44177e5f1b71-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.237526 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.237508 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28622389-8eda-400c-aca2-44177e5f1b71-proxy-tls\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.338474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.338450 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28622389-8eda-400c-aca2-44177e5f1b71-isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.338578 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.338478 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28622389-8eda-400c-aca2-44177e5f1b71-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.338578 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.338504 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28622389-8eda-400c-aca2-44177e5f1b71-proxy-tls\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.338578 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.338530 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmn4\" (UniqueName: \"kubernetes.io/projected/28622389-8eda-400c-aca2-44177e5f1b71-kube-api-access-fpmn4\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.338932 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.338906 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28622389-8eda-400c-aca2-44177e5f1b71-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.339241 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.339218 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28622389-8eda-400c-aca2-44177e5f1b71-isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.340873 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.340853 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28622389-8eda-400c-aca2-44177e5f1b71-proxy-tls\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.346661 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.346637 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmn4\" (UniqueName: \"kubernetes.io/projected/28622389-8eda-400c-aca2-44177e5f1b71-kube-api-access-fpmn4\") pod \"isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.477634 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.477581 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:16.606454 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.606405 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp"] May 25 13:20:16.608842 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:20:16.608817 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28622389_8eda_400c_aca2_44177e5f1b71.slice/crio-eb8eef050d0755aa18dd53e5b69937d666df4f200a5731e4c8bb4a7962bd0f16 WatchSource:0}: Error finding container eb8eef050d0755aa18dd53e5b69937d666df4f200a5731e4c8bb4a7962bd0f16: Status 404 returned error can't find the container with id eb8eef050d0755aa18dd53e5b69937d666df4f200a5731e4c8bb4a7962bd0f16 May 25 13:20:16.623358 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:16.623331 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerStarted","Data":"eb8eef050d0755aa18dd53e5b69937d666df4f200a5731e4c8bb4a7962bd0f16"} May 25 13:20:20.640761 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:20.640726 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerStarted","Data":"21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f"} May 25 13:20:24.656341 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:24.656297 2554 generic.go:358] "Generic (PLEG): container finished" podID="28622389-8eda-400c-aca2-44177e5f1b71" containerID="21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f" exitCode=0 May 25 13:20:24.656717 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:24.656369 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerDied","Data":"21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f"} May 25 13:20:37.711077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:37.711034 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerStarted","Data":"4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e"} May 25 13:20:39.720300 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:39.720230 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerStarted","Data":"8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f"} May 25 13:20:42.737049 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:42.737014 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerStarted","Data":"605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf"} May 25 13:20:42.737457 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:42.737274 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:42.737457 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:42.737397 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:42.738337 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:42.738302 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:20:42.763369 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:42.763329 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podStartSLOduration=1.610326409 podStartE2EDuration="26.763317407s" podCreationTimestamp="2026-05-25 13:20:16 +0000 UTC" firstStartedPulling="2026-05-25 13:20:16.611311682 +0000 UTC m=+828.289609277" lastFinishedPulling="2026-05-25 13:20:41.764302696 +0000 UTC m=+853.442600275" observedRunningTime="2026-05-25 13:20:42.761070372 +0000 UTC m=+854.439367974" watchObservedRunningTime="2026-05-25 13:20:42.763317407 +0000 UTC m=+854.441615009" May 25 13:20:43.741141 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:43.741108 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:43.741595 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:43.741261 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:20:43.742079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:43.742054 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:20:44.744089 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:44.744044 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:20:44.744589 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:44.744407 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:20:44.747528 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:44.747507 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:20:45.747280 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:45.747236 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:20:45.747675 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:45.747462 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:20:55.747162 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:55.747117 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:20:55.747847 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:20:55.747528 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:21:05.747253 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:05.747207 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:21:05.747710 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:05.747682 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:21:15.747711 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:15.747667 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:21:15.748246 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:15.748223 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:21:25.748098 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:25.748056 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:21:25.748608 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:25.748472 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:21:28.842680 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:28.842651 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:21:28.844988 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:28.844964 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:21:35.747661 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:35.747622 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:21:35.748143 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:35.748119 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:21:45.748141 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:45.748104 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:21:45.748585 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:21:45.748280 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:22:01.613677 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:01.613642 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp"] May 25 13:22:01.615091 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:01.613995 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" containerID="cri-o://4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e" gracePeriod=30 May 25 13:22:01.615091 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:01.614034 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" containerID="cri-o://605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf" gracePeriod=30 May 25 13:22:01.615091 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:01.614046 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" containerID="cri-o://8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f" gracePeriod=30 May 25 13:22:02.011041 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.010965 2554 generic.go:358] "Generic (PLEG): container finished" podID="28622389-8eda-400c-aca2-44177e5f1b71" containerID="8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f" exitCode=2 May 25 13:22:02.011175 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.011039 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerDied","Data":"8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f"} May 25 13:22:02.097833 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.097799 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t"] May 25 13:22:02.101065 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.101044 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.104262 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.104239 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\"" May 25 13:22:02.104939 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.104920 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a698c-predictor-serving-cert\"" May 25 13:22:02.119933 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.119911 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t"] May 25 13:22:02.152946 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.152912 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.152946 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.152956 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-proxy-tls\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.153097 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.152984 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.153097 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.153053 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766cl\" (UniqueName: \"kubernetes.io/projected/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kube-api-access-766cl\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.254042 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.254009 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.254200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.254048 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-proxy-tls\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.254200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.254074 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.254200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.254097 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-766cl\" (UniqueName: \"kubernetes.io/projected/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kube-api-access-766cl\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.254382 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.254359 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.254685 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.254666 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.256640 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.256614 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-proxy-tls\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.302208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.302151 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766cl\" (UniqueName: \"kubernetes.io/projected/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kube-api-access-766cl\") pod \"isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.411644 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.411612 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:02.556595 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:02.556523 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t"] May 25 13:22:02.559872 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:22:02.559841 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2f347b_2ab0_4e09_b055_f4d29cce5ecc.slice/crio-49b9b2682f68ac02d283402106afaed7fa4270f77b69d659fa1bc07507ae7e52 WatchSource:0}: Error finding container 49b9b2682f68ac02d283402106afaed7fa4270f77b69d659fa1bc07507ae7e52: Status 404 returned error can't find the container with id 49b9b2682f68ac02d283402106afaed7fa4270f77b69d659fa1bc07507ae7e52 May 25 13:22:03.016450 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:03.016392 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerStarted","Data":"cc1af6ba123e8ce1314e65322358c5ae036ad7cff77ac2ff851f6afd5650b25f"} May 25 13:22:03.016450 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:03.016447 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerStarted","Data":"49b9b2682f68ac02d283402106afaed7fa4270f77b69d659fa1bc07507ae7e52"} May 25 13:22:04.744408 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:04.744365 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 25 13:22:05.747258 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:05.747210 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:22:05.747672 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:05.747585 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:22:06.027997 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:06.027929 2554 generic.go:358] "Generic (PLEG): container finished" podID="28622389-8eda-400c-aca2-44177e5f1b71" containerID="4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e" exitCode=0 May 25 13:22:06.028124 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:06.028001 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerDied","Data":"4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e"} May 25 13:22:07.032167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:07.032089 2554 generic.go:358] "Generic (PLEG): container finished" podID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerID="cc1af6ba123e8ce1314e65322358c5ae036ad7cff77ac2ff851f6afd5650b25f" exitCode=0 May 25 13:22:07.032609 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:07.032163 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerDied","Data":"cc1af6ba123e8ce1314e65322358c5ae036ad7cff77ac2ff851f6afd5650b25f"} May 25 13:22:08.037981 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:08.037949 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerStarted","Data":"f041dd47aea7cd6230eb6ca37c44eb05bfae727c5c0bd309874924eb144eca90"} May 25 13:22:08.037981 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:08.037984 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerStarted","Data":"0044cf94e8da0b7a28943c8daadffeca37c2b6493617df5d2ac385b0d9dd5dd6"} May 25 13:22:08.038375 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:08.038179 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:08.074627 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:08.074582 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podStartSLOduration=6.074568389 podStartE2EDuration="6.074568389s" podCreationTimestamp="2026-05-25 13:22:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:22:08.072937181 +0000 UTC m=+939.751234809" watchObservedRunningTime="2026-05-25 13:22:08.074568389 +0000 UTC m=+939.752865990" May 25 13:22:09.041697 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:09.041666 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:09.042775 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:09.042743 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:22:09.744554 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:09.744516 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 25 13:22:10.044781 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:10.044695 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:22:14.744553 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:14.744513 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 25 13:22:14.744916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:14.744654 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:22:15.049619 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:15.049547 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:22:15.049945 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:15.049919 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:22:15.747163 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:15.747123 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:22:15.747542 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:15.747450 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:22:19.745148 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:19.745108 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 25 13:22:24.744533 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:24.744485 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 25 13:22:25.050843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:25.050765 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:22:25.748033 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:25.747977 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused" May 25 13:22:25.748420 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:25.748106 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:22:25.748420 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:25.748262 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:22:25.748420 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:25.748390 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:22:29.744660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:29.744612 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused" May 25 13:22:31.755588 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.755566 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:22:31.874280 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.874223 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpmn4\" (UniqueName: \"kubernetes.io/projected/28622389-8eda-400c-aca2-44177e5f1b71-kube-api-access-fpmn4\") pod \"28622389-8eda-400c-aca2-44177e5f1b71\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " May 25 13:22:31.874280 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.874264 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28622389-8eda-400c-aca2-44177e5f1b71-isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\") pod \"28622389-8eda-400c-aca2-44177e5f1b71\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " May 25 13:22:31.874463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.874335 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28622389-8eda-400c-aca2-44177e5f1b71-proxy-tls\") pod \"28622389-8eda-400c-aca2-44177e5f1b71\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " May 25 13:22:31.874463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.874392 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28622389-8eda-400c-aca2-44177e5f1b71-kserve-provision-location\") pod \"28622389-8eda-400c-aca2-44177e5f1b71\" (UID: \"28622389-8eda-400c-aca2-44177e5f1b71\") " May 25 13:22:31.874739 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.874705 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28622389-8eda-400c-aca2-44177e5f1b71-isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config") pod "28622389-8eda-400c-aca2-44177e5f1b71" (UID: "28622389-8eda-400c-aca2-44177e5f1b71"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:22:31.874739 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.874730 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28622389-8eda-400c-aca2-44177e5f1b71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28622389-8eda-400c-aca2-44177e5f1b71" (UID: "28622389-8eda-400c-aca2-44177e5f1b71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:22:31.876468 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.876440 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28622389-8eda-400c-aca2-44177e5f1b71-kube-api-access-fpmn4" (OuterVolumeSpecName: "kube-api-access-fpmn4") pod "28622389-8eda-400c-aca2-44177e5f1b71" (UID: "28622389-8eda-400c-aca2-44177e5f1b71"). InnerVolumeSpecName "kube-api-access-fpmn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:22:31.876468 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.876446 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28622389-8eda-400c-aca2-44177e5f1b71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28622389-8eda-400c-aca2-44177e5f1b71" (UID: "28622389-8eda-400c-aca2-44177e5f1b71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:22:31.975592 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.975570 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpmn4\" (UniqueName: \"kubernetes.io/projected/28622389-8eda-400c-aca2-44177e5f1b71-kube-api-access-fpmn4\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:22:31.975592 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.975590 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28622389-8eda-400c-aca2-44177e5f1b71-isvc-raw-sklearn-batcher-13cfe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:22:31.975717 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.975602 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28622389-8eda-400c-aca2-44177e5f1b71-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:22:31.975717 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:31.975613 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28622389-8eda-400c-aca2-44177e5f1b71-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:22:32.120791 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.120755 2554 generic.go:358] "Generic (PLEG): container finished" podID="28622389-8eda-400c-aca2-44177e5f1b71" containerID="605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf" exitCode=0 May 25 13:22:32.120918 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.120853 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" May 25 13:22:32.120918 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.120865 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerDied","Data":"605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf"} May 25 13:22:32.120918 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.120900 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp" event={"ID":"28622389-8eda-400c-aca2-44177e5f1b71","Type":"ContainerDied","Data":"eb8eef050d0755aa18dd53e5b69937d666df4f200a5731e4c8bb4a7962bd0f16"} May 25 13:22:32.121018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.120920 2554 scope.go:117] "RemoveContainer" containerID="605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf" May 25 13:22:32.130133 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.130114 2554 scope.go:117] "RemoveContainer" containerID="8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f" May 25 13:22:32.139032 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.139016 2554 scope.go:117] "RemoveContainer" containerID="4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e" May 25 13:22:32.143826 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.143801 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp"] May 25 13:22:32.147171 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.147146 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-13cfe-predictor-7dcf74b8f7-vhgrp"] May 25 13:22:32.148197 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.148177 2554 scope.go:117] "RemoveContainer" containerID="21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f" May 25 13:22:32.154742 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.154723 2554 scope.go:117] "RemoveContainer" containerID="605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf" May 25 13:22:32.154968 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:22:32.154949 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf\": container with ID starting with 605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf not found: ID does not exist" containerID="605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf" May 25 13:22:32.155047 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.154980 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf"} err="failed to get container status \"605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf\": rpc error: code = NotFound desc = could not find container \"605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf\": container with ID starting with 605ea59707c397d82940e1c73fe4bf1c5b4b75cfb6021da197a138b965f66aaf not found: ID does not exist" May 25 13:22:32.155047 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.155005 2554 scope.go:117] "RemoveContainer" containerID="8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f" May 25 13:22:32.155255 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:22:32.155240 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f\": container with ID starting with 8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f not found: ID does not exist" containerID="8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f" May 25 13:22:32.155294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.155260 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f"} err="failed to get container status \"8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f\": rpc error: code = NotFound desc = could not find container \"8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f\": container with ID starting with 8e557b712fc37a05c977c416d8b8e66998883cf0aefd69a7eac8f5ae8630752f not found: ID does not exist" May 25 13:22:32.155294 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.155274 2554 scope.go:117] "RemoveContainer" containerID="4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e" May 25 13:22:32.155504 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:22:32.155484 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e\": container with ID starting with 4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e not found: ID does not exist" containerID="4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e" May 25 13:22:32.155577 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.155513 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e"} err="failed to get container status \"4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e\": rpc error: code = NotFound desc = could not find container \"4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e\": container with ID starting with 4da6b58d89a46a243db11c155c2f9d28f09881772ef624b5519eea98e174dc0e not found: ID does not exist" May 25 13:22:32.155577 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.155533 2554 scope.go:117] "RemoveContainer" containerID="21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f" May 25 13:22:32.155761 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:22:32.155745 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f\": container with ID starting with 21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f not found: ID does not exist" containerID="21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f" May 25 13:22:32.155800 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.155764 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f"} err="failed to get container status \"21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f\": rpc error: code = NotFound desc = could not find container \"21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f\": container with ID starting with 21b08a01622a2a9a49d2e8945e2c52aa4d76188aa781e5a7e00c93981976812f not found: ID does not exist" May 25 13:22:32.900648 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:32.900615 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28622389-8eda-400c-aca2-44177e5f1b71" path="/var/lib/kubelet/pods/28622389-8eda-400c-aca2-44177e5f1b71/volumes" May 25 13:22:35.050456 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:35.050400 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:22:45.050452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:45.050340 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:22:55.050237 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:22:55.050194 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:23:05.050309 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:05.050270 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:23:15.050574 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:15.050544 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:23:41.783063 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.783029 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t"] May 25 13:23:41.783562 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.783452 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" containerID="cri-o://0044cf94e8da0b7a28943c8daadffeca37c2b6493617df5d2ac385b0d9dd5dd6" gracePeriod=30 May 25 13:23:41.783562 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.783486 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kube-rbac-proxy" containerID="cri-o://f041dd47aea7cd6230eb6ca37c44eb05bfae727c5c0bd309874924eb144eca90" gracePeriod=30 May 25 13:23:41.864789 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.864758 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls"] May 25 13:23:41.865120 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865107 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" May 25 13:23:41.865167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865121 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" May 25 13:23:41.865167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865136 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" May 25 13:23:41.865167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865142 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" May 25 13:23:41.865167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865150 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="storage-initializer" May 25 13:23:41.865167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865155 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="storage-initializer" May 25 13:23:41.865167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865167 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" May 25 13:23:41.865346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865172 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" May 25 13:23:41.865346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865221 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="agent" May 25 13:23:41.865346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865229 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kube-rbac-proxy" May 25 13:23:41.865346 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.865237 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="28622389-8eda-400c-aca2-44177e5f1b71" containerName="kserve-container" May 25 13:23:41.867983 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.867966 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:41.870957 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.870937 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-12a34-predictor-serving-cert\"" May 25 13:23:41.871157 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.871132 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\"" May 25 13:23:41.878969 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.878950 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls"] May 25 13:23:41.915312 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.915284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2622473-c13e-4c4e-b185-23d2b032c6a6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:41.915406 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.915383 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbwxn\" (UniqueName: \"kubernetes.io/projected/a2622473-c13e-4c4e-b185-23d2b032c6a6-kube-api-access-qbwxn\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:41.915474 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.915420 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2622473-c13e-4c4e-b185-23d2b032c6a6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:41.915512 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:41.915492 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2622473-c13e-4c4e-b185-23d2b032c6a6-isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.016089 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.016051 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qbwxn\" (UniqueName: \"kubernetes.io/projected/a2622473-c13e-4c4e-b185-23d2b032c6a6-kube-api-access-qbwxn\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.016285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.016099 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2622473-c13e-4c4e-b185-23d2b032c6a6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.016285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.016138 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2622473-c13e-4c4e-b185-23d2b032c6a6-isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.016285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.016213 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2622473-c13e-4c4e-b185-23d2b032c6a6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.016741 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.016717 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2622473-c13e-4c4e-b185-23d2b032c6a6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.017030 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.017001 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2622473-c13e-4c4e-b185-23d2b032c6a6-isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.018710 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.018690 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2622473-c13e-4c4e-b185-23d2b032c6a6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.027154 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.027130 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbwxn\" (UniqueName: \"kubernetes.io/projected/a2622473-c13e-4c4e-b185-23d2b032c6a6-kube-api-access-qbwxn\") pod \"isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.180142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.180118 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:42.310725 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.310701 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls"] May 25 13:23:42.312609 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:23:42.312585 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2622473_c13e_4c4e_b185_23d2b032c6a6.slice/crio-27fb6e5e46ed1340d43340ad9ac7467db741c725bf4fa5c5d25e8e20fffc74bc WatchSource:0}: Error finding container 27fb6e5e46ed1340d43340ad9ac7467db741c725bf4fa5c5d25e8e20fffc74bc: Status 404 returned error can't find the container with id 27fb6e5e46ed1340d43340ad9ac7467db741c725bf4fa5c5d25e8e20fffc74bc May 25 13:23:42.314463 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.314447 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 13:23:42.365754 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.365732 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerStarted","Data":"27fb6e5e46ed1340d43340ad9ac7467db741c725bf4fa5c5d25e8e20fffc74bc"} May 25 13:23:42.367567 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.367544 2554 generic.go:358] "Generic (PLEG): container finished" podID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerID="f041dd47aea7cd6230eb6ca37c44eb05bfae727c5c0bd309874924eb144eca90" exitCode=2 May 25 13:23:42.367673 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:42.367601 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerDied","Data":"f041dd47aea7cd6230eb6ca37c44eb05bfae727c5c0bd309874924eb144eca90"} May 25 13:23:43.373055 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:43.373018 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerStarted","Data":"bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5"} May 25 13:23:45.045303 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.045258 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" May 25 13:23:45.050651 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.050624 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" May 25 13:23:45.383562 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.383528 2554 generic.go:358] "Generic (PLEG): container finished" podID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerID="0044cf94e8da0b7a28943c8daadffeca37c2b6493617df5d2ac385b0d9dd5dd6" exitCode=0 May 25 13:23:45.383688 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.383592 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerDied","Data":"0044cf94e8da0b7a28943c8daadffeca37c2b6493617df5d2ac385b0d9dd5dd6"} May 25 13:23:45.426034 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.426009 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:23:45.548072 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.548049 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kserve-provision-location\") pod \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " May 25 13:23:45.548180 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.548102 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-766cl\" (UniqueName: \"kubernetes.io/projected/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kube-api-access-766cl\") pod \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " May 25 13:23:45.548180 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.548130 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\") pod \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " May 25 13:23:45.548180 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.548148 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-proxy-tls\") pod \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\" (UID: \"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc\") " May 25 13:23:45.548419 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.548391 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" (UID: "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:23:45.548539 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.548505 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config") pod "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" (UID: "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc"). InnerVolumeSpecName "isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:23:45.550214 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.550161 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" (UID: "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:23:45.550285 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.550234 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kube-api-access-766cl" (OuterVolumeSpecName: "kube-api-access-766cl") pod "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" (UID: "4e2f347b-2ab0-4e09-b055-f4d29cce5ecc"). InnerVolumeSpecName "kube-api-access-766cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:23:45.649067 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.649043 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-isvc-sklearn-graph-raw-a698c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:23:45.649067 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.649064 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:23:45.649191 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.649074 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:23:45.649191 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:45.649083 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766cl\" (UniqueName: \"kubernetes.io/projected/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc-kube-api-access-766cl\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:23:46.395200 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.395160 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" event={"ID":"4e2f347b-2ab0-4e09-b055-f4d29cce5ecc","Type":"ContainerDied","Data":"49b9b2682f68ac02d283402106afaed7fa4270f77b69d659fa1bc07507ae7e52"} May 25 13:23:46.395614 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.395209 2554 scope.go:117] "RemoveContainer" containerID="f041dd47aea7cd6230eb6ca37c44eb05bfae727c5c0bd309874924eb144eca90" May 25 13:23:46.395614 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.395173 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t" May 25 13:23:46.396730 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.396703 2554 generic.go:358] "Generic (PLEG): container finished" podID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerID="bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5" exitCode=0 May 25 13:23:46.396870 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.396760 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerDied","Data":"bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5"} May 25 13:23:46.404011 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.403990 2554 scope.go:117] "RemoveContainer" containerID="0044cf94e8da0b7a28943c8daadffeca37c2b6493617df5d2ac385b0d9dd5dd6" May 25 13:23:46.411464 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.411444 2554 scope.go:117] "RemoveContainer" containerID="cc1af6ba123e8ce1314e65322358c5ae036ad7cff77ac2ff851f6afd5650b25f" May 25 13:23:46.435779 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.435757 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t"] May 25 13:23:46.440090 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.440071 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a698c-predictor-869c5879d5-x5k6t"] May 25 13:23:46.902522 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:46.902491 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" path="/var/lib/kubelet/pods/4e2f347b-2ab0-4e09-b055-f4d29cce5ecc/volumes" May 25 13:23:47.402387 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:47.402350 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerStarted","Data":"e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597"} May 25 13:23:47.402387 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:47.402388 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerStarted","Data":"3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6"} May 25 13:23:47.402843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:47.402688 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:47.402843 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:47.402832 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:47.403888 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:47.403866 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:23:47.424677 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:47.424628 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podStartSLOduration=6.424612756 podStartE2EDuration="6.424612756s" podCreationTimestamp="2026-05-25 13:23:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:23:47.422279845 +0000 UTC m=+1039.100577491" watchObservedRunningTime="2026-05-25 13:23:47.424612756 +0000 UTC m=+1039.102910359" May 25 13:23:48.405757 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:48.405714 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:23:53.410244 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:53.410212 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:23:53.410831 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:23:53.410804 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:24:03.410834 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:24:03.410793 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:24:13.411274 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:24:13.411230 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:24:23.411327 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:24:23.411283 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:24:33.410775 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:24:33.410735 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:24:43.411648 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:24:43.411611 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:24:53.411634 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:24:53.411598 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:25:22.091774 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:22.091742 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls"] May 25 13:25:22.092192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:22.092039 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" containerID="cri-o://3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6" gracePeriod=30 May 25 13:25:22.092192 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:22.092074 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kube-rbac-proxy" containerID="cri-o://e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597" gracePeriod=30 May 25 13:25:22.739985 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:22.739948 2554 generic.go:358] "Generic (PLEG): container finished" podID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerID="e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597" exitCode=2 May 25 13:25:22.740129 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:22.740022 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerDied","Data":"e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597"} May 25 13:25:23.406339 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:23.406296 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused" May 25 13:25:23.410994 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:23.410970 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused" May 25 13:25:25.545019 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.544998 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:25:25.689783 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.689710 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2622473-c13e-4c4e-b185-23d2b032c6a6-proxy-tls\") pod \"a2622473-c13e-4c4e-b185-23d2b032c6a6\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " May 25 13:25:25.689783 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.689769 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2622473-c13e-4c4e-b185-23d2b032c6a6-kserve-provision-location\") pod \"a2622473-c13e-4c4e-b185-23d2b032c6a6\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " May 25 13:25:25.689947 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.689805 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbwxn\" (UniqueName: \"kubernetes.io/projected/a2622473-c13e-4c4e-b185-23d2b032c6a6-kube-api-access-qbwxn\") pod \"a2622473-c13e-4c4e-b185-23d2b032c6a6\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " May 25 13:25:25.689947 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.689829 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2622473-c13e-4c4e-b185-23d2b032c6a6-isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\") pod \"a2622473-c13e-4c4e-b185-23d2b032c6a6\" (UID: \"a2622473-c13e-4c4e-b185-23d2b032c6a6\") " May 25 13:25:25.690146 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.690122 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2622473-c13e-4c4e-b185-23d2b032c6a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2622473-c13e-4c4e-b185-23d2b032c6a6" (UID: "a2622473-c13e-4c4e-b185-23d2b032c6a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:25:25.690219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.690195 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2622473-c13e-4c4e-b185-23d2b032c6a6-isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config") pod "a2622473-c13e-4c4e-b185-23d2b032c6a6" (UID: "a2622473-c13e-4c4e-b185-23d2b032c6a6"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:25:25.691839 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.691817 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2622473-c13e-4c4e-b185-23d2b032c6a6-kube-api-access-qbwxn" (OuterVolumeSpecName: "kube-api-access-qbwxn") pod "a2622473-c13e-4c4e-b185-23d2b032c6a6" (UID: "a2622473-c13e-4c4e-b185-23d2b032c6a6"). InnerVolumeSpecName "kube-api-access-qbwxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:25:25.691975 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.691958 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2622473-c13e-4c4e-b185-23d2b032c6a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2622473-c13e-4c4e-b185-23d2b032c6a6" (UID: "a2622473-c13e-4c4e-b185-23d2b032c6a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:25:25.751308 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.751282 2554 generic.go:358] "Generic (PLEG): container finished" podID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerID="3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6" exitCode=0 May 25 13:25:25.751407 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.751357 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" May 25 13:25:25.751407 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.751369 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerDied","Data":"3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6"} May 25 13:25:25.751522 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.751409 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls" event={"ID":"a2622473-c13e-4c4e-b185-23d2b032c6a6","Type":"ContainerDied","Data":"27fb6e5e46ed1340d43340ad9ac7467db741c725bf4fa5c5d25e8e20fffc74bc"} May 25 13:25:25.751522 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.751442 2554 scope.go:117] "RemoveContainer" containerID="e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597" May 25 13:25:25.761695 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.761676 2554 scope.go:117] "RemoveContainer" containerID="3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6" May 25 13:25:25.769519 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.769475 2554 scope.go:117] "RemoveContainer" containerID="bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5" May 25 13:25:25.776021 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.775999 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls"] May 25 13:25:25.776570 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.776548 2554 scope.go:117] "RemoveContainer" containerID="e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597" May 25 13:25:25.776822 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:25:25.776803 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597\": container with ID starting with e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597 not found: ID does not exist" containerID="e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597" May 25 13:25:25.776900 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.776836 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597"} err="failed to get container status \"e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597\": rpc error: code = NotFound desc = could not find container \"e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597\": container with ID starting with e628c0dff35d7bd768a3752ac365a2550e427bbb330c0c485a880a30171bb597 not found: ID does not exist" May 25 13:25:25.776900 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.776859 2554 scope.go:117] "RemoveContainer" containerID="3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6" May 25 13:25:25.777102 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:25:25.777085 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6\": container with ID starting with 3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6 not found: ID does not exist" containerID="3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6" May 25 13:25:25.777145 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.777116 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6"} err="failed to get container status \"3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6\": rpc error: code = NotFound desc = could not find container \"3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6\": container with ID starting with 3feb992b745647fa076cbc922889ab7d7516cc180272da5ab39f20414f7fd7f6 not found: ID does not exist" May 25 13:25:25.777145 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.777134 2554 scope.go:117] "RemoveContainer" containerID="bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5" May 25 13:25:25.777369 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:25:25.777353 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5\": container with ID starting with bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5 not found: ID does not exist" containerID="bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5" May 25 13:25:25.777473 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.777380 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5"} err="failed to get container status \"bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5\": rpc error: code = NotFound desc = could not find container \"bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5\": container with ID starting with bb216cab2b0c72f19ca478aff2d6e07441feb78b8f3f461b1c4a7a471f14eef5 not found: ID does not exist" May 25 13:25:25.779765 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.779745 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-12a34-predictor-5c75494b9c-665ls"] May 25 13:25:25.791207 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.791189 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2622473-c13e-4c4e-b185-23d2b032c6a6-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:25:25.791263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.791210 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2622473-c13e-4c4e-b185-23d2b032c6a6-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:25:25.791263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.791221 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbwxn\" (UniqueName: \"kubernetes.io/projected/a2622473-c13e-4c4e-b185-23d2b032c6a6-kube-api-access-qbwxn\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:25:25.791263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:25.791231 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2622473-c13e-4c4e-b185-23d2b032c6a6-isvc-sklearn-graph-raw-hpa-12a34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:25:26.901394 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:26.901353 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" path="/var/lib/kubelet/pods/a2622473-c13e-4c4e-b185-23d2b032c6a6/volumes" May 25 13:25:42.173232 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173157 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8"] May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173518 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="storage-initializer" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173530 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="storage-initializer" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173545 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="storage-initializer" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173550 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="storage-initializer" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173556 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173562 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173568 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kube-rbac-proxy" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173573 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kube-rbac-proxy" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173580 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kube-rbac-proxy" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173585 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kube-rbac-proxy" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173596 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" May 25 13:25:42.173623 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173601 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" May 25 13:25:42.173968 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173652 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kserve-container" May 25 13:25:42.173968 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173662 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kube-rbac-proxy" May 25 13:25:42.173968 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173668 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2622473-c13e-4c4e-b185-23d2b032c6a6" containerName="kube-rbac-proxy" May 25 13:25:42.173968 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.173676 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e2f347b-2ab0-4e09-b055-f4d29cce5ecc" containerName="kserve-container" May 25 13:25:42.178734 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.178715 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.181570 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.181548 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-df494-predictor-serving-cert\"" May 25 13:25:42.181715 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.181646 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-df494-kube-rbac-proxy-sar-config\"" May 25 13:25:42.181899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.181885 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmtw7\"" May 25 13:25:42.181952 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.181929 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" May 25 13:25:42.183177 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.183160 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" May 25 13:25:42.187732 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.187708 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8"] May 25 13:25:42.315037 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.315009 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-df494-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8398bcec-949e-41ca-adaa-cdd0e919a256-isvc-logger-raw-df494-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.315138 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.315061 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.315138 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.315086 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kgb\" (UniqueName: \"kubernetes.io/projected/8398bcec-949e-41ca-adaa-cdd0e919a256-kube-api-access-d4kgb\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.315211 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.315143 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8398bcec-949e-41ca-adaa-cdd0e919a256-kserve-provision-location\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.415724 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.415693 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.415989 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.415732 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kgb\" (UniqueName: \"kubernetes.io/projected/8398bcec-949e-41ca-adaa-cdd0e919a256-kube-api-access-d4kgb\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.415989 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.415756 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8398bcec-949e-41ca-adaa-cdd0e919a256-kserve-provision-location\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.415989 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.415804 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-df494-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8398bcec-949e-41ca-adaa-cdd0e919a256-isvc-logger-raw-df494-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.415989 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:25:42.415874 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-serving-cert: secret "isvc-logger-raw-df494-predictor-serving-cert" not found May 25 13:25:42.415989 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:25:42.415961 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls podName:8398bcec-949e-41ca-adaa-cdd0e919a256 nodeName:}" failed. No retries permitted until 2026-05-25 13:25:42.915936894 +0000 UTC m=+1154.594234520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls") pod "isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" (UID: "8398bcec-949e-41ca-adaa-cdd0e919a256") : secret "isvc-logger-raw-df494-predictor-serving-cert" not found May 25 13:25:42.416208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.416136 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8398bcec-949e-41ca-adaa-cdd0e919a256-kserve-provision-location\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.416393 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.416375 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-df494-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8398bcec-949e-41ca-adaa-cdd0e919a256-isvc-logger-raw-df494-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.424610 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.424558 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kgb\" (UniqueName: \"kubernetes.io/projected/8398bcec-949e-41ca-adaa-cdd0e919a256-kube-api-access-d4kgb\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.918512 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.918484 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:42.920890 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:42.920861 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls\") pod \"isvc-logger-raw-df494-predictor-8568b6779f-zxpw8\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:43.090028 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:43.090001 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:43.220018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:43.219990 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8"] May 25 13:25:43.222606 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:25:43.222577 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8398bcec_949e_41ca_adaa_cdd0e919a256.slice/crio-a488a9effc23f50edd0a0d287f59a06490943c0bc1029063ba3d5e0df419927f WatchSource:0}: Error finding container a488a9effc23f50edd0a0d287f59a06490943c0bc1029063ba3d5e0df419927f: Status 404 returned error can't find the container with id a488a9effc23f50edd0a0d287f59a06490943c0bc1029063ba3d5e0df419927f May 25 13:25:43.819706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:43.819664 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerStarted","Data":"7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b"} May 25 13:25:43.819706 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:43.819707 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerStarted","Data":"a488a9effc23f50edd0a0d287f59a06490943c0bc1029063ba3d5e0df419927f"} May 25 13:25:47.837869 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:47.837833 2554 generic.go:358] "Generic (PLEG): container finished" podID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerID="7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b" exitCode=0 May 25 13:25:47.838229 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:47.837908 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerDied","Data":"7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b"} May 25 13:25:48.843908 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:48.843875 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerStarted","Data":"89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904"} May 25 13:25:48.843908 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:48.843912 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerStarted","Data":"d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38"} May 25 13:25:48.844438 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:48.843921 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerStarted","Data":"980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c"} May 25 13:25:48.844438 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:48.844143 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:48.866459 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:48.866408 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podStartSLOduration=6.866395512 podStartE2EDuration="6.866395512s" podCreationTimestamp="2026-05-25 13:25:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:25:48.864625513 +0000 UTC m=+1160.542923117" watchObservedRunningTime="2026-05-25 13:25:48.866395512 +0000 UTC m=+1160.544693113" May 25 13:25:49.848225 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:49.848138 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:49.848225 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:49.848179 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:49.849685 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:49.849647 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:25:49.850300 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:49.850276 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:25:50.851443 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:50.851390 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:25:50.851871 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:50.851743 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:25:55.856056 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:55.856026 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:25:55.856638 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:55.856606 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:25:55.856875 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:25:55.856847 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:26:05.856601 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:05.856561 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:26:05.857148 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:05.857054 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:26:15.856585 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:15.856544 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:26:15.857073 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:15.857021 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:26:25.856631 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:25.856592 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:26:25.857127 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:25.857075 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:26:28.871753 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:28.871723 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:26:28.874510 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:28.874489 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:26:35.857183 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:35.857134 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:26:35.857643 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:35.857619 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:26:45.857096 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:45.857058 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:26:45.857632 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:45.857602 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:26:55.857401 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:55.857363 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:26:55.857993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:26:55.857672 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:27:07.369481 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.369450 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8"] May 25 13:27:07.369993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.369808 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" containerID="cri-o://980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c" gracePeriod=30 May 25 13:27:07.369993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.369880 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" containerID="cri-o://d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38" gracePeriod=30 May 25 13:27:07.369993 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.369878 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" containerID="cri-o://89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904" gracePeriod=30 May 25 13:27:07.416265 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.416243 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl"] May 25 13:27:07.419770 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.419756 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.422029 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.422009 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-df600-predictor-serving-cert\"" May 25 13:27:07.422137 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.422097 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\"" May 25 13:27:07.428367 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.428349 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl"] May 25 13:27:07.553725 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.553695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rcmd\" (UniqueName: \"kubernetes.io/projected/7492cd80-e2d2-474d-b94a-816b3a8736ba-kube-api-access-7rcmd\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.553866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.553734 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7492cd80-e2d2-474d-b94a-816b3a8736ba-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.553866 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.553820 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.553966 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.553932 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7492cd80-e2d2-474d-b94a-816b3a8736ba-isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.655023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.654956 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7492cd80-e2d2-474d-b94a-816b3a8736ba-isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.655023 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.654993 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcmd\" (UniqueName: \"kubernetes.io/projected/7492cd80-e2d2-474d-b94a-816b3a8736ba-kube-api-access-7rcmd\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.655193 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.655036 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7492cd80-e2d2-474d-b94a-816b3a8736ba-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.655193 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.655093 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.655296 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:27:07.655258 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-serving-cert: secret "isvc-sklearn-scale-raw-df600-predictor-serving-cert" not found May 25 13:27:07.655353 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:27:07.655333 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls podName:7492cd80-e2d2-474d-b94a-816b3a8736ba nodeName:}" failed. No retries permitted until 2026-05-25 13:27:08.155313112 +0000 UTC m=+1239.833610711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls") pod "isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" (UID: "7492cd80-e2d2-474d-b94a-816b3a8736ba") : secret "isvc-sklearn-scale-raw-df600-predictor-serving-cert" not found May 25 13:27:07.655416 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.655383 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7492cd80-e2d2-474d-b94a-816b3a8736ba-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.655662 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.655642 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7492cd80-e2d2-474d-b94a-816b3a8736ba-isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:07.663280 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:07.663262 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcmd\" (UniqueName: \"kubernetes.io/projected/7492cd80-e2d2-474d-b94a-816b3a8736ba-kube-api-access-7rcmd\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:08.121228 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:08.121200 2554 generic.go:358] "Generic (PLEG): container finished" podID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerID="d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38" exitCode=2 May 25 13:27:08.121388 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:08.121276 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerDied","Data":"d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38"} May 25 13:27:08.157772 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:08.157748 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:08.160064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:08.160041 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls\") pod \"isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:08.331079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:08.331044 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:08.456696 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:08.456670 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl"] May 25 13:27:08.458643 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:27:08.458616 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7492cd80_e2d2_474d_b94a_816b3a8736ba.slice/crio-92133ad4c2c2577e46b684d0b381b29f683dd2ac17c6f096908bc44fb8f02b77 WatchSource:0}: Error finding container 92133ad4c2c2577e46b684d0b381b29f683dd2ac17c6f096908bc44fb8f02b77: Status 404 returned error can't find the container with id 92133ad4c2c2577e46b684d0b381b29f683dd2ac17c6f096908bc44fb8f02b77 May 25 13:27:09.126337 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:09.126301 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerStarted","Data":"b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb"} May 25 13:27:09.126337 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:09.126338 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerStarted","Data":"92133ad4c2c2577e46b684d0b381b29f683dd2ac17c6f096908bc44fb8f02b77"} May 25 13:27:10.852468 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:10.852409 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused" May 25 13:27:11.135555 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:11.135528 2554 generic.go:358] "Generic (PLEG): container finished" podID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerID="980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c" exitCode=0 May 25 13:27:11.135691 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:11.135591 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerDied","Data":"980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c"} May 25 13:27:13.143300 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:13.143264 2554 generic.go:358] "Generic (PLEG): container finished" podID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerID="b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb" exitCode=0 May 25 13:27:13.143670 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:13.143337 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerDied","Data":"b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb"} May 25 13:27:14.149437 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:14.149383 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerStarted","Data":"cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1"} May 25 13:27:14.149840 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:14.149443 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerStarted","Data":"8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422"} May 25 13:27:14.149840 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:14.149750 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:14.149932 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:14.149873 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:14.151191 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:14.151164 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:27:14.170683 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:14.170640 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podStartSLOduration=7.170629018 podStartE2EDuration="7.170629018s" podCreationTimestamp="2026-05-25 13:27:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:27:14.169322583 +0000 UTC m=+1245.847620196" watchObservedRunningTime="2026-05-25 13:27:14.170629018 +0000 UTC m=+1245.848926621" May 25 13:27:15.153174 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:15.153132 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:27:15.851526 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:15.851483 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused" May 25 13:27:15.856833 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:15.856807 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:27:15.857206 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:15.857176 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:27:20.158085 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:20.158057 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:27:20.158562 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:20.158535 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:27:20.851460 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:20.851399 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused" May 25 13:27:20.851627 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:20.851542 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:27:25.852496 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:25.852456 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused" May 25 13:27:25.857464 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:25.857409 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:27:25.857801 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:25.857774 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:27:30.158868 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:30.158830 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:27:30.851725 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:30.851678 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused" May 25 13:27:35.852532 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:35.852472 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused" May 25 13:27:35.856824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:35.856791 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused" May 25 13:27:35.856970 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:35.856951 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:27:35.857234 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:35.857198 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 25 13:27:35.857365 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:35.857349 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:27:37.520527 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.520503 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:27:37.676544 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.676470 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls\") pod \"8398bcec-949e-41ca-adaa-cdd0e919a256\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " May 25 13:27:37.676544 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.676530 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8398bcec-949e-41ca-adaa-cdd0e919a256-kserve-provision-location\") pod \"8398bcec-949e-41ca-adaa-cdd0e919a256\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " May 25 13:27:37.676744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.676573 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-df494-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8398bcec-949e-41ca-adaa-cdd0e919a256-isvc-logger-raw-df494-kube-rbac-proxy-sar-config\") pod \"8398bcec-949e-41ca-adaa-cdd0e919a256\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " May 25 13:27:37.676744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.676622 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4kgb\" (UniqueName: \"kubernetes.io/projected/8398bcec-949e-41ca-adaa-cdd0e919a256-kube-api-access-d4kgb\") pod \"8398bcec-949e-41ca-adaa-cdd0e919a256\" (UID: \"8398bcec-949e-41ca-adaa-cdd0e919a256\") " May 25 13:27:37.676897 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.676827 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8398bcec-949e-41ca-adaa-cdd0e919a256-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8398bcec-949e-41ca-adaa-cdd0e919a256" (UID: "8398bcec-949e-41ca-adaa-cdd0e919a256"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:27:37.676950 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.676924 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8398bcec-949e-41ca-adaa-cdd0e919a256-isvc-logger-raw-df494-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-df494-kube-rbac-proxy-sar-config") pod "8398bcec-949e-41ca-adaa-cdd0e919a256" (UID: "8398bcec-949e-41ca-adaa-cdd0e919a256"). InnerVolumeSpecName "isvc-logger-raw-df494-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:27:37.678496 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.678475 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8398bcec-949e-41ca-adaa-cdd0e919a256" (UID: "8398bcec-949e-41ca-adaa-cdd0e919a256"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:27:37.678691 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.678676 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8398bcec-949e-41ca-adaa-cdd0e919a256-kube-api-access-d4kgb" (OuterVolumeSpecName: "kube-api-access-d4kgb") pod "8398bcec-949e-41ca-adaa-cdd0e919a256" (UID: "8398bcec-949e-41ca-adaa-cdd0e919a256"). InnerVolumeSpecName "kube-api-access-d4kgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:27:37.777865 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.777841 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kgb\" (UniqueName: \"kubernetes.io/projected/8398bcec-949e-41ca-adaa-cdd0e919a256-kube-api-access-d4kgb\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:27:37.777865 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.777863 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8398bcec-949e-41ca-adaa-cdd0e919a256-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:27:37.777990 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.777874 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8398bcec-949e-41ca-adaa-cdd0e919a256-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:27:37.777990 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:37.777886 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-df494-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8398bcec-949e-41ca-adaa-cdd0e919a256-isvc-logger-raw-df494-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:27:38.238283 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.238251 2554 generic.go:358] "Generic (PLEG): container finished" podID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerID="89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904" exitCode=0 May 25 13:27:38.238409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.238324 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerDied","Data":"89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904"} May 25 13:27:38.238409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.238357 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" event={"ID":"8398bcec-949e-41ca-adaa-cdd0e919a256","Type":"ContainerDied","Data":"a488a9effc23f50edd0a0d287f59a06490943c0bc1029063ba3d5e0df419927f"} May 25 13:27:38.238409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.238358 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8" May 25 13:27:38.238409 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.238372 2554 scope.go:117] "RemoveContainer" containerID="89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904" May 25 13:27:38.247420 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.247402 2554 scope.go:117] "RemoveContainer" containerID="d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38" May 25 13:27:38.254518 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.254504 2554 scope.go:117] "RemoveContainer" containerID="980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c" May 25 13:27:38.260886 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.260864 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8"] May 25 13:27:38.261873 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.261857 2554 scope.go:117] "RemoveContainer" containerID="7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b" May 25 13:27:38.266974 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.266948 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df494-predictor-8568b6779f-zxpw8"] May 25 13:27:38.269279 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.269264 2554 scope.go:117] "RemoveContainer" containerID="89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904" May 25 13:27:38.269557 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:27:38.269537 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904\": container with ID starting with 89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904 not found: ID does not exist" containerID="89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904" May 25 13:27:38.269620 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.269564 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904"} err="failed to get container status \"89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904\": rpc error: code = NotFound desc = could not find container \"89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904\": container with ID starting with 89c302a55fd634c11dd7e127169421178027d859cef387e0f558773f76e87904 not found: ID does not exist" May 25 13:27:38.269620 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.269582 2554 scope.go:117] "RemoveContainer" containerID="d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38" May 25 13:27:38.269816 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:27:38.269798 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38\": container with ID starting with d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38 not found: ID does not exist" containerID="d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38" May 25 13:27:38.269865 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.269822 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38"} err="failed to get container status \"d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38\": rpc error: code = NotFound desc = could not find container \"d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38\": container with ID starting with d8a864f3fba1e369e28b1f60ad2fb90b8b8af5a3b06db9b14d78382ddfd5fd38 not found: ID does not exist" May 25 13:27:38.269865 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.269838 2554 scope.go:117] "RemoveContainer" containerID="980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c" May 25 13:27:38.270054 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:27:38.270033 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c\": container with ID starting with 980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c not found: ID does not exist" containerID="980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c" May 25 13:27:38.270135 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.270061 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c"} err="failed to get container status \"980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c\": rpc error: code = NotFound desc = could not find container \"980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c\": container with ID starting with 980f9cc5da2d2ff445c29d7cb869f960d11cd6c5e2b226a0a72238e2c511ce4c not found: ID does not exist" May 25 13:27:38.270135 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.270082 2554 scope.go:117] "RemoveContainer" containerID="7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b" May 25 13:27:38.270291 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:27:38.270276 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b\": container with ID starting with 7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b not found: ID does not exist" containerID="7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b" May 25 13:27:38.270343 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.270298 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b"} err="failed to get container status \"7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b\": rpc error: code = NotFound desc = could not find container \"7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b\": container with ID starting with 7edb8a3d0f8ba4b84f1d88e6846f5f38d9d5a945a2ea9c427a0818a79d9fbf1b not found: ID does not exist" May 25 13:27:38.901550 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:38.901518 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" path="/var/lib/kubelet/pods/8398bcec-949e-41ca-adaa-cdd0e919a256/volumes" May 25 13:27:40.159229 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:40.159193 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:27:50.158472 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:27:50.158413 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:28:00.158535 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:28:00.158494 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:28:10.158664 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:28:10.158621 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:28:20.159138 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:28:20.159098 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:28:30.158545 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:28:30.158503 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:28:40.158592 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:28:40.158510 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:28:47.897052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:28:47.897008 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:28:57.896984 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:28:57.896933 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:29:07.897208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:07.897149 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:29:17.898052 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:17.898016 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:29:27.602476 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.602441 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl"] May 25 13:29:27.602846 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.602744 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" containerID="cri-o://8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422" gracePeriod=30 May 25 13:29:27.602846 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.602783 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kube-rbac-proxy" containerID="cri-o://cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1" gracePeriod=30 May 25 13:29:27.752887 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.752859 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv"] May 25 13:29:27.753211 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753200 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" May 25 13:29:27.753263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753214 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" May 25 13:29:27.753263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753222 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" May 25 13:29:27.753263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753227 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" May 25 13:29:27.753263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753236 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" May 25 13:29:27.753263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753241 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" May 25 13:29:27.753263 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753263 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="storage-initializer" May 25 13:29:27.753472 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753268 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="storage-initializer" May 25 13:29:27.753472 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753324 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="agent" May 25 13:29:27.753472 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753334 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kube-rbac-proxy" May 25 13:29:27.753472 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.753343 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="8398bcec-949e-41ca-adaa-cdd0e919a256" containerName="kserve-container" May 25 13:29:27.756397 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.756380 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.763117 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.763095 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6e8942-predictor-serving-cert\"" May 25 13:29:27.763219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.763105 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6e8942-kube-rbac-proxy-sar-config\"" May 25 13:29:27.769839 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.769818 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv"] May 25 13:29:27.788686 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.788667 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.788794 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.788694 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b036bdc-7c13-4c49-a651-45f69c09b52a-kserve-provision-location\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.788794 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.788720 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b036bdc-7c13-4c49-a651-45f69c09b52a-isvc-primary-6e8942-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.788794 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.788743 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wblz\" (UniqueName: \"kubernetes.io/projected/5b036bdc-7c13-4c49-a651-45f69c09b52a-kube-api-access-8wblz\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.889679 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.889628 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.889679 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.889657 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b036bdc-7c13-4c49-a651-45f69c09b52a-kserve-provision-location\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.889837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.889681 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b036bdc-7c13-4c49-a651-45f69c09b52a-isvc-primary-6e8942-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.889837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.889705 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8wblz\" (UniqueName: \"kubernetes.io/projected/5b036bdc-7c13-4c49-a651-45f69c09b52a-kube-api-access-8wblz\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.889837 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:29:27.889799 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-6e8942-predictor-serving-cert: secret "isvc-primary-6e8942-predictor-serving-cert" not found May 25 13:29:27.889999 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:29:27.889878 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls podName:5b036bdc-7c13-4c49-a651-45f69c09b52a nodeName:}" failed. No retries permitted until 2026-05-25 13:29:28.389858676 +0000 UTC m=+1380.068156258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls") pod "isvc-primary-6e8942-predictor-6f55cf5856-wbktv" (UID: "5b036bdc-7c13-4c49-a651-45f69c09b52a") : secret "isvc-primary-6e8942-predictor-serving-cert" not found May 25 13:29:27.890094 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.890071 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b036bdc-7c13-4c49-a651-45f69c09b52a-kserve-provision-location\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.890321 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.890302 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b036bdc-7c13-4c49-a651-45f69c09b52a-isvc-primary-6e8942-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:27.897048 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.897022 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused" May 25 13:29:27.900123 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:27.900105 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wblz\" (UniqueName: \"kubernetes.io/projected/5b036bdc-7c13-4c49-a651-45f69c09b52a-kube-api-access-8wblz\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:28.393476 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:28.393442 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:28.395823 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:28.395802 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls\") pod \"isvc-primary-6e8942-predictor-6f55cf5856-wbktv\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:28.631620 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:28.631591 2554 generic.go:358] "Generic (PLEG): container finished" podID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerID="cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1" exitCode=2 May 25 13:29:28.631967 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:28.631669 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerDied","Data":"cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1"} May 25 13:29:28.666959 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:28.666901 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:28.796751 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:28.796617 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv"] May 25 13:29:28.799126 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:29:28.799089 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b036bdc_7c13_4c49_a651_45f69c09b52a.slice/crio-1417669b0d8bed7933f2c2ebb1220ff515b59477641cda48ffe7428b8f3370f0 WatchSource:0}: Error finding container 1417669b0d8bed7933f2c2ebb1220ff515b59477641cda48ffe7428b8f3370f0: Status 404 returned error can't find the container with id 1417669b0d8bed7933f2c2ebb1220ff515b59477641cda48ffe7428b8f3370f0 May 25 13:29:28.801003 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:28.800988 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 13:29:29.636812 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:29.636768 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerStarted","Data":"21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3"} May 25 13:29:29.637154 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:29.636818 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerStarted","Data":"1417669b0d8bed7933f2c2ebb1220ff515b59477641cda48ffe7428b8f3370f0"} May 25 13:29:30.153618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:30.153576 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" May 25 13:29:32.650209 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:32.650176 2554 generic.go:358] "Generic (PLEG): container finished" podID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerID="21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3" exitCode=0 May 25 13:29:32.650681 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:32.650240 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerDied","Data":"21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3"} May 25 13:29:33.655702 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:33.655671 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerStarted","Data":"015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797"} May 25 13:29:33.656066 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:33.655713 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerStarted","Data":"59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9"} May 25 13:29:33.656066 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:33.655927 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:34.659665 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:34.659632 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:34.660713 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:34.660683 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:29:35.154308 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:35.154263 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused" May 25 13:29:35.662690 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:35.662644 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:29:36.342324 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.342298 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:29:36.369088 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.369037 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podStartSLOduration=9.369019336 podStartE2EDuration="9.369019336s" podCreationTimestamp="2026-05-25 13:29:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:29:33.675849862 +0000 UTC m=+1385.354147500" watchObservedRunningTime="2026-05-25 13:29:36.369019336 +0000 UTC m=+1388.047316938" May 25 13:29:36.459899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.459838 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rcmd\" (UniqueName: \"kubernetes.io/projected/7492cd80-e2d2-474d-b94a-816b3a8736ba-kube-api-access-7rcmd\") pod \"7492cd80-e2d2-474d-b94a-816b3a8736ba\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " May 25 13:29:36.459899 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.459871 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7492cd80-e2d2-474d-b94a-816b3a8736ba-kserve-provision-location\") pod \"7492cd80-e2d2-474d-b94a-816b3a8736ba\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " May 25 13:29:36.460044 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.459907 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7492cd80-e2d2-474d-b94a-816b3a8736ba-isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\") pod \"7492cd80-e2d2-474d-b94a-816b3a8736ba\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " May 25 13:29:36.460044 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.459948 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls\") pod \"7492cd80-e2d2-474d-b94a-816b3a8736ba\" (UID: \"7492cd80-e2d2-474d-b94a-816b3a8736ba\") " May 25 13:29:36.460167 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.460145 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7492cd80-e2d2-474d-b94a-816b3a8736ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7492cd80-e2d2-474d-b94a-816b3a8736ba" (UID: "7492cd80-e2d2-474d-b94a-816b3a8736ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:29:36.460271 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.460249 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7492cd80-e2d2-474d-b94a-816b3a8736ba-isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config") pod "7492cd80-e2d2-474d-b94a-816b3a8736ba" (UID: "7492cd80-e2d2-474d-b94a-816b3a8736ba"). InnerVolumeSpecName "isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:29:36.461957 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.461937 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7492cd80-e2d2-474d-b94a-816b3a8736ba-kube-api-access-7rcmd" (OuterVolumeSpecName: "kube-api-access-7rcmd") pod "7492cd80-e2d2-474d-b94a-816b3a8736ba" (UID: "7492cd80-e2d2-474d-b94a-816b3a8736ba"). InnerVolumeSpecName "kube-api-access-7rcmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:29:36.462144 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.462119 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7492cd80-e2d2-474d-b94a-816b3a8736ba" (UID: "7492cd80-e2d2-474d-b94a-816b3a8736ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:29:36.560724 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.560696 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7492cd80-e2d2-474d-b94a-816b3a8736ba-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:29:36.560724 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.560721 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rcmd\" (UniqueName: \"kubernetes.io/projected/7492cd80-e2d2-474d-b94a-816b3a8736ba-kube-api-access-7rcmd\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:29:36.560875 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.560735 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7492cd80-e2d2-474d-b94a-816b3a8736ba-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:29:36.560875 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.560749 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7492cd80-e2d2-474d-b94a-816b3a8736ba-isvc-sklearn-scale-raw-df600-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:29:36.667105 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.667078 2554 generic.go:358] "Generic (PLEG): container finished" podID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerID="8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422" exitCode=0 May 25 13:29:36.667465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.667129 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerDied","Data":"8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422"} May 25 13:29:36.667465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.667160 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" event={"ID":"7492cd80-e2d2-474d-b94a-816b3a8736ba","Type":"ContainerDied","Data":"92133ad4c2c2577e46b684d0b381b29f683dd2ac17c6f096908bc44fb8f02b77"} May 25 13:29:36.667465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.667161 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl" May 25 13:29:36.667465 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.667175 2554 scope.go:117] "RemoveContainer" containerID="cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1" May 25 13:29:36.676162 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.676144 2554 scope.go:117] "RemoveContainer" containerID="8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422" May 25 13:29:36.683267 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.683249 2554 scope.go:117] "RemoveContainer" containerID="b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb" May 25 13:29:36.692363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.690226 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl"] May 25 13:29:36.693668 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.693647 2554 scope.go:117] "RemoveContainer" containerID="cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1" May 25 13:29:36.693972 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:29:36.693942 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1\": container with ID starting with cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1 not found: ID does not exist" containerID="cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1" May 25 13:29:36.694062 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.693983 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1"} err="failed to get container status \"cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1\": rpc error: code = NotFound desc = could not find container \"cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1\": container with ID starting with cbcfda2a5f697258f3a7cb97b5a1a674f6904b38d5c59da32293b6569e6722f1 not found: ID does not exist" May 25 13:29:36.694062 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.694008 2554 scope.go:117] "RemoveContainer" containerID="8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422" May 25 13:29:36.694278 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:29:36.694259 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422\": container with ID starting with 8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422 not found: ID does not exist" containerID="8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422" May 25 13:29:36.694341 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.694286 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422"} err="failed to get container status \"8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422\": rpc error: code = NotFound desc = could not find container \"8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422\": container with ID starting with 8ae022ba73b36eccb380fcd68fac613f3f30dcdbcb2825a7b49a12549fc00422 not found: ID does not exist" May 25 13:29:36.694341 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.694305 2554 scope.go:117] "RemoveContainer" containerID="b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb" May 25 13:29:36.694590 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:29:36.694566 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb\": container with ID starting with b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb not found: ID does not exist" containerID="b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb" May 25 13:29:36.694660 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.694599 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb"} err="failed to get container status \"b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb\": rpc error: code = NotFound desc = could not find container \"b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb\": container with ID starting with b38eee75830f066ef3462f4a0b19e5c87dd34bde9fdfd5e1854c35acac2dabcb not found: ID does not exist" May 25 13:29:36.695529 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.695511 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-df600-predictor-6bd77fdbc5-2xpsl"] May 25 13:29:36.901141 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:36.901119 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" path="/var/lib/kubelet/pods/7492cd80-e2d2-474d-b94a-816b3a8736ba/volumes" May 25 13:29:40.666394 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:40.666367 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:29:40.666971 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:40.666944 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:29:50.667752 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:29:50.667714 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:30:00.667480 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:00.667441 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:30:10.666919 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:10.666838 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:30:20.666895 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:20.666848 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:30:30.666948 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:30.666904 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused" May 25 13:30:40.667628 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:40.667595 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:30:47.872593 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.872554 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96"] May 25 13:30:47.872958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.872898 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" May 25 13:30:47.872958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.872909 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" May 25 13:30:47.872958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.872918 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kube-rbac-proxy" May 25 13:30:47.872958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.872923 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kube-rbac-proxy" May 25 13:30:47.872958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.872938 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="storage-initializer" May 25 13:30:47.872958 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.872944 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="storage-initializer" May 25 13:30:47.873149 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.873004 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kserve-container" May 25 13:30:47.873149 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.873014 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="7492cd80-e2d2-474d-b94a-816b3a8736ba" containerName="kube-rbac-proxy" May 25 13:30:47.876223 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.876206 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.880377 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.880352 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" May 25 13:30:47.880524 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.880354 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-6e8942\"" May 25 13:30:47.880524 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.880399 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-6e8942-dockercfg-6jnzl\"" May 25 13:30:47.881029 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.881013 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6e8942-predictor-serving-cert\"" May 25 13:30:47.881073 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.881014 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6e8942-kube-rbac-proxy-sar-config\"" May 25 13:30:47.897808 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.897791 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96"] May 25 13:30:47.898018 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.897997 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-proxy-tls\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.898088 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.898029 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kserve-provision-location\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.898088 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.898051 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-isvc-secondary-6e8942-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.898190 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.898138 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxv9\" (UniqueName: \"kubernetes.io/projected/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kube-api-access-wtxv9\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.898190 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.898174 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-cabundle-cert\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.999271 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.999247 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-proxy-tls\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.999367 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.999276 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kserve-provision-location\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.999367 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.999307 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-isvc-secondary-6e8942-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.999367 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.999358 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxv9\" (UniqueName: \"kubernetes.io/projected/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kube-api-access-wtxv9\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.999557 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.999389 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-cabundle-cert\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:47.999981 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:47.999957 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kserve-provision-location\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:48.000077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.000013 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-isvc-secondary-6e8942-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:48.000077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.000029 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-cabundle-cert\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:48.001710 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.001692 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-proxy-tls\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:48.008221 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.008198 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxv9\" (UniqueName: \"kubernetes.io/projected/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kube-api-access-wtxv9\") pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:48.186556 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.186495 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:30:48.314810 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.314779 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96"] May 25 13:30:48.317479 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:30:48.317449 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46f6084_3a1a_4a0e_bb91_da18dd0048ec.slice/crio-d079202f112eff8bfa2b9a01c3199c28931d8b370de4acf89f3f5f68b7b0e6fc WatchSource:0}: Error finding container d079202f112eff8bfa2b9a01c3199c28931d8b370de4acf89f3f5f68b7b0e6fc: Status 404 returned error can't find the container with id d079202f112eff8bfa2b9a01c3199c28931d8b370de4acf89f3f5f68b7b0e6fc May 25 13:30:48.921747 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.921712 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" event={"ID":"f46f6084-3a1a-4a0e-bb91-da18dd0048ec","Type":"ContainerStarted","Data":"9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09"} May 25 13:30:48.922141 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:48.921757 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" event={"ID":"f46f6084-3a1a-4a0e-bb91-da18dd0048ec","Type":"ContainerStarted","Data":"d079202f112eff8bfa2b9a01c3199c28931d8b370de4acf89f3f5f68b7b0e6fc"} May 25 13:30:53.940201 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:53.940172 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e8942-predictor-6bc776f855-frj96_f46f6084-3a1a-4a0e-bb91-da18dd0048ec/storage-initializer/0.log" May 25 13:30:53.940601 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:53.940213 2554 generic.go:358] "Generic (PLEG): container finished" podID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerID="9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09" exitCode=1 May 25 13:30:53.940601 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:53.940294 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" event={"ID":"f46f6084-3a1a-4a0e-bb91-da18dd0048ec","Type":"ContainerDied","Data":"9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09"} May 25 13:30:54.945079 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:54.945048 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e8942-predictor-6bc776f855-frj96_f46f6084-3a1a-4a0e-bb91-da18dd0048ec/storage-initializer/0.log" May 25 13:30:54.945493 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:30:54.945109 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" event={"ID":"f46f6084-3a1a-4a0e-bb91-da18dd0048ec","Type":"ContainerStarted","Data":"3419d4fc5665f4b8bf71515cdc2247d4bdfa817091c8e5fcb91a51bc50bc409d"} May 25 13:31:00.966578 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:00.966549 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e8942-predictor-6bc776f855-frj96_f46f6084-3a1a-4a0e-bb91-da18dd0048ec/storage-initializer/1.log" May 25 13:31:00.966978 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:00.966866 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e8942-predictor-6bc776f855-frj96_f46f6084-3a1a-4a0e-bb91-da18dd0048ec/storage-initializer/0.log" May 25 13:31:00.966978 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:00.966896 2554 generic.go:358] "Generic (PLEG): container finished" podID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerID="3419d4fc5665f4b8bf71515cdc2247d4bdfa817091c8e5fcb91a51bc50bc409d" exitCode=1 May 25 13:31:00.967053 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:00.966979 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" event={"ID":"f46f6084-3a1a-4a0e-bb91-da18dd0048ec","Type":"ContainerDied","Data":"3419d4fc5665f4b8bf71515cdc2247d4bdfa817091c8e5fcb91a51bc50bc409d"} May 25 13:31:00.967053 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:00.967029 2554 scope.go:117] "RemoveContainer" containerID="9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09" May 25 13:31:00.967404 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:00.967388 2554 scope.go:117] "RemoveContainer" containerID="9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09" May 25 13:31:00.978610 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:00.978582 2554 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6e8942-predictor-6bc776f855-frj96_kserve-ci-e2e-test_f46f6084-3a1a-4a0e-bb91-da18dd0048ec_0 in pod sandbox d079202f112eff8bfa2b9a01c3199c28931d8b370de4acf89f3f5f68b7b0e6fc from index: no such id: '9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09'" containerID="9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09" May 25 13:31:00.978681 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:00.978633 2554 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6e8942-predictor-6bc776f855-frj96_kserve-ci-e2e-test_f46f6084-3a1a-4a0e-bb91-da18dd0048ec_0 in pod sandbox d079202f112eff8bfa2b9a01c3199c28931d8b370de4acf89f3f5f68b7b0e6fc from index: no such id: '9ffce288c3fbaf02c676ae8efe51051f9495d270d33c23adc653c3a489835a09'; Skipping pod \"isvc-secondary-6e8942-predictor-6bc776f855-frj96_kserve-ci-e2e-test(f46f6084-3a1a-4a0e-bb91-da18dd0048ec)\"" logger="UnhandledError" May 25 13:31:00.979940 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:00.979919 2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-6e8942-predictor-6bc776f855-frj96_kserve-ci-e2e-test(f46f6084-3a1a-4a0e-bb91-da18dd0048ec)\"" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" May 25 13:31:01.971354 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:01.971326 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e8942-predictor-6bc776f855-frj96_f46f6084-3a1a-4a0e-bb91-da18dd0048ec/storage-initializer/1.log" May 25 13:31:05.916443 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:05.916399 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96"] May 25 13:31:05.938128 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:05.938098 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv"] May 25 13:31:05.938442 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:05.938386 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" containerID="cri-o://59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9" gracePeriod=30 May 25 13:31:05.938570 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:05.938501 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kube-rbac-proxy" containerID="cri-o://015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797" gracePeriod=30 May 25 13:31:06.058003 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.057983 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e8942-predictor-6bc776f855-frj96_f46f6084-3a1a-4a0e-bb91-da18dd0048ec/storage-initializer/1.log" May 25 13:31:06.058098 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.058050 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:31:06.134662 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.134635 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kserve-provision-location\") pod \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " May 25 13:31:06.134803 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.134692 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-isvc-secondary-6e8942-kube-rbac-proxy-sar-config\") pod \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " May 25 13:31:06.134803 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.134750 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-proxy-tls\") pod \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " May 25 13:31:06.134803 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.134786 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-cabundle-cert\") pod \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " May 25 13:31:06.134970 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.134917 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtxv9\" (UniqueName: \"kubernetes.io/projected/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kube-api-access-wtxv9\") pod \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\" (UID: \"f46f6084-3a1a-4a0e-bb91-da18dd0048ec\") " May 25 13:31:06.135028 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.134954 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f46f6084-3a1a-4a0e-bb91-da18dd0048ec" (UID: "f46f6084-3a1a-4a0e-bb91-da18dd0048ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:31:06.135110 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.135092 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f46f6084-3a1a-4a0e-bb91-da18dd0048ec" (UID: "f46f6084-3a1a-4a0e-bb91-da18dd0048ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:31:06.135153 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.135124 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-isvc-secondary-6e8942-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-6e8942-kube-rbac-proxy-sar-config") pod "f46f6084-3a1a-4a0e-bb91-da18dd0048ec" (UID: "f46f6084-3a1a-4a0e-bb91-da18dd0048ec"). InnerVolumeSpecName "isvc-secondary-6e8942-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:31:06.135286 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.135265 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:06.135286 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.135285 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-isvc-secondary-6e8942-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:06.135459 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.135297 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-cabundle-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:06.136721 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.136700 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f46f6084-3a1a-4a0e-bb91-da18dd0048ec" (UID: "f46f6084-3a1a-4a0e-bb91-da18dd0048ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:31:06.136925 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.136909 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kube-api-access-wtxv9" (OuterVolumeSpecName: "kube-api-access-wtxv9") pod "f46f6084-3a1a-4a0e-bb91-da18dd0048ec" (UID: "f46f6084-3a1a-4a0e-bb91-da18dd0048ec"). InnerVolumeSpecName "kube-api-access-wtxv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:31:06.176496 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.176443 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7"] May 25 13:31:06.176827 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.176815 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerName="storage-initializer" May 25 13:31:06.176872 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.176828 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerName="storage-initializer" May 25 13:31:06.176872 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.176835 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerName="storage-initializer" May 25 13:31:06.176872 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.176841 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerName="storage-initializer" May 25 13:31:06.176964 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.176910 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerName="storage-initializer" May 25 13:31:06.176964 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.176919 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" containerName="storage-initializer" May 25 13:31:06.180370 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.180355 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.182701 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.182661 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-b18b30\"" May 25 13:31:06.182884 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.182870 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b18b30-predictor-serving-cert\"" May 25 13:31:06.183048 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.183028 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\"" May 25 13:31:06.183588 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.183568 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-b18b30-dockercfg-gdj2k\"" May 25 13:31:06.191693 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.191673 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7"] May 25 13:31:06.236398 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.236379 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2z2x\" (UniqueName: \"kubernetes.io/projected/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kube-api-access-l2z2x\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.236505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.236408 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kserve-provision-location\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.236505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.236485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-cabundle-cert\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.236583 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.236514 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-proxy-tls\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.236618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.236583 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.236653 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.236644 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:06.236688 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.236659 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxv9\" (UniqueName: \"kubernetes.io/projected/f46f6084-3a1a-4a0e-bb91-da18dd0048ec-kube-api-access-wtxv9\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:06.337556 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.337529 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-cabundle-cert\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.337662 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.337566 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-proxy-tls\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.337662 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.337594 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.337662 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.337629 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l2z2x\" (UniqueName: \"kubernetes.io/projected/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kube-api-access-l2z2x\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.337662 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.337647 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kserve-provision-location\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.338084 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.338065 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kserve-provision-location\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.338183 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.338135 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-cabundle-cert\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.338279 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.338259 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.339932 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.339915 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-proxy-tls\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.346001 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.345982 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2z2x\" (UniqueName: \"kubernetes.io/projected/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kube-api-access-l2z2x\") pod \"isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.490769 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.490714 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:06.624744 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.624718 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7"] May 25 13:31:06.627117 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:31:06.627087 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1cb364_7dd6_4875_ae28_bcae03d9ac59.slice/crio-48cdea0e34e5f19f685216f7e3873315b6f3bcd2d27e9e2fa46c968a97bea884 WatchSource:0}: Error finding container 48cdea0e34e5f19f685216f7e3873315b6f3bcd2d27e9e2fa46c968a97bea884: Status 404 returned error can't find the container with id 48cdea0e34e5f19f685216f7e3873315b6f3bcd2d27e9e2fa46c968a97bea884 May 25 13:31:06.990233 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.990196 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" event={"ID":"5c1cb364-7dd6-4875-ae28-bcae03d9ac59","Type":"ContainerStarted","Data":"9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4"} May 25 13:31:06.990671 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.990240 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" event={"ID":"5c1cb364-7dd6-4875-ae28-bcae03d9ac59","Type":"ContainerStarted","Data":"48cdea0e34e5f19f685216f7e3873315b6f3bcd2d27e9e2fa46c968a97bea884"} May 25 13:31:06.992134 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.992112 2554 generic.go:358] "Generic (PLEG): container finished" podID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerID="015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797" exitCode=2 May 25 13:31:06.992250 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.992180 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerDied","Data":"015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797"} May 25 13:31:06.993338 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.993315 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e8942-predictor-6bc776f855-frj96_f46f6084-3a1a-4a0e-bb91-da18dd0048ec/storage-initializer/1.log" May 25 13:31:06.993447 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.993413 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" event={"ID":"f46f6084-3a1a-4a0e-bb91-da18dd0048ec","Type":"ContainerDied","Data":"d079202f112eff8bfa2b9a01c3199c28931d8b370de4acf89f3f5f68b7b0e6fc"} May 25 13:31:06.993501 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.993471 2554 scope.go:117] "RemoveContainer" containerID="3419d4fc5665f4b8bf71515cdc2247d4bdfa817091c8e5fcb91a51bc50bc409d" May 25 13:31:06.993540 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:06.993434 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96" May 25 13:31:07.054665 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:07.054632 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96"] May 25 13:31:07.058111 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:07.058086 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e8942-predictor-6bc776f855-frj96"] May 25 13:31:08.902482 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:08.902448 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46f6084-3a1a-4a0e-bb91-da18dd0048ec" path="/var/lib/kubelet/pods/f46f6084-3a1a-4a0e-bb91-da18dd0048ec/volumes" May 25 13:31:09.683994 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.683970 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:31:09.767554 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.767487 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls\") pod \"5b036bdc-7c13-4c49-a651-45f69c09b52a\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " May 25 13:31:09.767554 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.767524 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wblz\" (UniqueName: \"kubernetes.io/projected/5b036bdc-7c13-4c49-a651-45f69c09b52a-kube-api-access-8wblz\") pod \"5b036bdc-7c13-4c49-a651-45f69c09b52a\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " May 25 13:31:09.767705 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.767577 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b036bdc-7c13-4c49-a651-45f69c09b52a-isvc-primary-6e8942-kube-rbac-proxy-sar-config\") pod \"5b036bdc-7c13-4c49-a651-45f69c09b52a\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " May 25 13:31:09.767705 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.767602 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b036bdc-7c13-4c49-a651-45f69c09b52a-kserve-provision-location\") pod \"5b036bdc-7c13-4c49-a651-45f69c09b52a\" (UID: \"5b036bdc-7c13-4c49-a651-45f69c09b52a\") " May 25 13:31:09.767984 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.767950 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b036bdc-7c13-4c49-a651-45f69c09b52a-isvc-primary-6e8942-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-6e8942-kube-rbac-proxy-sar-config") pod "5b036bdc-7c13-4c49-a651-45f69c09b52a" (UID: "5b036bdc-7c13-4c49-a651-45f69c09b52a"). InnerVolumeSpecName "isvc-primary-6e8942-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:31:09.768111 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.768004 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b036bdc-7c13-4c49-a651-45f69c09b52a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b036bdc-7c13-4c49-a651-45f69c09b52a" (UID: "5b036bdc-7c13-4c49-a651-45f69c09b52a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:31:09.769444 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.769406 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b036bdc-7c13-4c49-a651-45f69c09b52a" (UID: "5b036bdc-7c13-4c49-a651-45f69c09b52a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:31:09.769558 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.769541 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b036bdc-7c13-4c49-a651-45f69c09b52a-kube-api-access-8wblz" (OuterVolumeSpecName: "kube-api-access-8wblz") pod "5b036bdc-7c13-4c49-a651-45f69c09b52a" (UID: "5b036bdc-7c13-4c49-a651-45f69c09b52a"). InnerVolumeSpecName "kube-api-access-8wblz". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:31:09.868607 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.868584 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b036bdc-7c13-4c49-a651-45f69c09b52a-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:09.868607 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.868605 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wblz\" (UniqueName: \"kubernetes.io/projected/5b036bdc-7c13-4c49-a651-45f69c09b52a-kube-api-access-8wblz\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:09.868730 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.868615 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-6e8942-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b036bdc-7c13-4c49-a651-45f69c09b52a-isvc-primary-6e8942-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:09.868730 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:09.868625 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b036bdc-7c13-4c49-a651-45f69c09b52a-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:10.006509 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.006489 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7_5c1cb364-7dd6-4875-ae28-bcae03d9ac59/storage-initializer/0.log" May 25 13:31:10.006842 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.006524 2554 generic.go:358] "Generic (PLEG): container finished" podID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerID="9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4" exitCode=1 May 25 13:31:10.006842 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.006599 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" event={"ID":"5c1cb364-7dd6-4875-ae28-bcae03d9ac59","Type":"ContainerDied","Data":"9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4"} May 25 13:31:10.008342 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.008318 2554 generic.go:358] "Generic (PLEG): container finished" podID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerID="59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9" exitCode=0 May 25 13:31:10.008452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.008360 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerDied","Data":"59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9"} May 25 13:31:10.008452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.008385 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" event={"ID":"5b036bdc-7c13-4c49-a651-45f69c09b52a","Type":"ContainerDied","Data":"1417669b0d8bed7933f2c2ebb1220ff515b59477641cda48ffe7428b8f3370f0"} May 25 13:31:10.008452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.008404 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv" May 25 13:31:10.008452 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.008407 2554 scope.go:117] "RemoveContainer" containerID="015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797" May 25 13:31:10.016638 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.016614 2554 scope.go:117] "RemoveContainer" containerID="59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9" May 25 13:31:10.024223 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.024205 2554 scope.go:117] "RemoveContainer" containerID="21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3" May 25 13:31:10.032541 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.032519 2554 scope.go:117] "RemoveContainer" containerID="015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797" May 25 13:31:10.032890 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:10.032871 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797\": container with ID starting with 015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797 not found: ID does not exist" containerID="015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797" May 25 13:31:10.032971 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.032897 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797"} err="failed to get container status \"015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797\": rpc error: code = NotFound desc = could not find container \"015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797\": container with ID starting with 015fcc76f8c24f8caf9810f75360737de275ce1cbd02920acee2ab699fda4797 not found: ID does not exist" May 25 13:31:10.032971 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.032913 2554 scope.go:117] "RemoveContainer" containerID="59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9" May 25 13:31:10.033147 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:10.033131 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9\": container with ID starting with 59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9 not found: ID does not exist" containerID="59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9" May 25 13:31:10.033208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.033149 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9"} err="failed to get container status \"59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9\": rpc error: code = NotFound desc = could not find container \"59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9\": container with ID starting with 59f26a712da4789de9aa542df73054c4d9c636e4bd137ea4033ea10ea11ff5f9 not found: ID does not exist" May 25 13:31:10.033208 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.033162 2554 scope.go:117] "RemoveContainer" containerID="21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3" May 25 13:31:10.033388 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:10.033369 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3\": container with ID starting with 21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3 not found: ID does not exist" containerID="21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3" May 25 13:31:10.033493 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.033393 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3"} err="failed to get container status \"21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3\": rpc error: code = NotFound desc = could not find container \"21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3\": container with ID starting with 21994f4fbc7a25fbd93457456acd95b23ab4ef1b136e6bcdaec836e0d0728ba3 not found: ID does not exist" May 25 13:31:10.044962 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.044942 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv"] May 25 13:31:10.052077 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.052056 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e8942-predictor-6f55cf5856-wbktv"] May 25 13:31:10.901369 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:10.901331 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" path="/var/lib/kubelet/pods/5b036bdc-7c13-4c49-a651-45f69c09b52a/volumes" May 25 13:31:11.014150 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.014128 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7_5c1cb364-7dd6-4875-ae28-bcae03d9ac59/storage-initializer/0.log" May 25 13:31:11.014515 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.014187 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" event={"ID":"5c1cb364-7dd6-4875-ae28-bcae03d9ac59","Type":"ContainerStarted","Data":"853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb"} May 25 13:31:11.014565 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.014522 2554 kubelet_pods.go:1019] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" secret="" err="secret \"fail-s3-sa-b18b30-dockercfg-gdj2k\" not found" May 25 13:31:11.155416 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.155340 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7"] May 25 13:31:11.388085 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388054 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs"] May 25 13:31:11.388447 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388415 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" May 25 13:31:11.388447 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388441 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" May 25 13:31:11.388548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388461 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="storage-initializer" May 25 13:31:11.388548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388467 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="storage-initializer" May 25 13:31:11.388548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388474 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kube-rbac-proxy" May 25 13:31:11.388548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388480 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kube-rbac-proxy" May 25 13:31:11.388548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388536 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kserve-container" May 25 13:31:11.388548 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.388547 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b036bdc-7c13-4c49-a651-45f69c09b52a" containerName="kube-rbac-proxy" May 25 13:31:11.391799 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.391784 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.399967 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.399944 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-a52ab-predictor-serving-cert\"" May 25 13:31:11.400476 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.400463 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmtw7\"" May 25 13:31:11.412366 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.412309 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs"] May 25 13:31:11.414402 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.414384 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-a52ab-kube-rbac-proxy-sar-config\"" May 25 13:31:11.482000 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.481979 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.482108 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.482020 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0250f8-e52a-433c-a0df-6a914c07e576-kserve-provision-location\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.482108 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.482083 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-a52ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae0250f8-e52a-433c-a0df-6a914c07e576-raw-sklearn-a52ab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.482188 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.482154 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcsz4\" (UniqueName: \"kubernetes.io/projected/ae0250f8-e52a-433c-a0df-6a914c07e576-kube-api-access-kcsz4\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.583073 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.583038 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kcsz4\" (UniqueName: \"kubernetes.io/projected/ae0250f8-e52a-433c-a0df-6a914c07e576-kube-api-access-kcsz4\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.583162 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.583088 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.583217 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:11.583203 2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-serving-cert: secret "raw-sklearn-a52ab-predictor-serving-cert" not found May 25 13:31:11.583269 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:11.583259 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls podName:ae0250f8-e52a-433c-a0df-6a914c07e576 nodeName:}" failed. No retries permitted until 2026-05-25 13:31:12.083241446 +0000 UTC m=+1483.761539026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls") pod "raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" (UID: "ae0250f8-e52a-433c-a0df-6a914c07e576") : secret "raw-sklearn-a52ab-predictor-serving-cert" not found May 25 13:31:11.583317 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.583283 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0250f8-e52a-433c-a0df-6a914c07e576-kserve-provision-location\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.583355 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.583330 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-a52ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae0250f8-e52a-433c-a0df-6a914c07e576-raw-sklearn-a52ab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.583704 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.583684 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0250f8-e52a-433c-a0df-6a914c07e576-kserve-provision-location\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.583945 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.583928 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-a52ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae0250f8-e52a-433c-a0df-6a914c07e576-raw-sklearn-a52ab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:11.596005 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:11.595978 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcsz4\" (UniqueName: \"kubernetes.io/projected/ae0250f8-e52a-433c-a0df-6a914c07e576-kube-api-access-kcsz4\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:12.017952 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:12.017916 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerName="storage-initializer" containerID="cri-o://853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb" gracePeriod=30 May 25 13:31:12.088304 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:12.088276 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:12.090637 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:12.090614 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls\") pod \"raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:12.301676 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:12.301617 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:12.438695 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:12.437807 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs"] May 25 13:31:12.442902 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:31:12.442878 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0250f8_e52a_433c_a0df_6a914c07e576.slice/crio-355353c1d3d03551140e6dc2eb9b604057d995ad0dd3dab72068654cffdd6697 WatchSource:0}: Error finding container 355353c1d3d03551140e6dc2eb9b604057d995ad0dd3dab72068654cffdd6697: Status 404 returned error can't find the container with id 355353c1d3d03551140e6dc2eb9b604057d995ad0dd3dab72068654cffdd6697 May 25 13:31:13.022279 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:13.022240 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerStarted","Data":"7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7"} May 25 13:31:13.022279 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:13.022279 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerStarted","Data":"355353c1d3d03551140e6dc2eb9b604057d995ad0dd3dab72068654cffdd6697"} May 25 13:31:14.268070 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.268046 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7_5c1cb364-7dd6-4875-ae28-bcae03d9ac59/storage-initializer/1.log" May 25 13:31:14.268481 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.268464 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7_5c1cb364-7dd6-4875-ae28-bcae03d9ac59/storage-initializer/0.log" May 25 13:31:14.268561 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.268547 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:14.406545 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406479 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\") pod \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " May 25 13:31:14.406545 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406519 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-proxy-tls\") pod \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " May 25 13:31:14.406722 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406572 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-cabundle-cert\") pod \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " May 25 13:31:14.406722 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406605 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kserve-provision-location\") pod \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " May 25 13:31:14.406722 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406623 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2z2x\" (UniqueName: \"kubernetes.io/projected/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kube-api-access-l2z2x\") pod \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\" (UID: \"5c1cb364-7dd6-4875-ae28-bcae03d9ac59\") " May 25 13:31:14.406927 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406903 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-isvc-init-fail-b18b30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-b18b30-kube-rbac-proxy-sar-config") pod "5c1cb364-7dd6-4875-ae28-bcae03d9ac59" (UID: "5c1cb364-7dd6-4875-ae28-bcae03d9ac59"). InnerVolumeSpecName "isvc-init-fail-b18b30-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:31:14.407000 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406903 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c1cb364-7dd6-4875-ae28-bcae03d9ac59" (UID: "5c1cb364-7dd6-4875-ae28-bcae03d9ac59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:31:14.407000 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.406982 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c1cb364-7dd6-4875-ae28-bcae03d9ac59" (UID: "5c1cb364-7dd6-4875-ae28-bcae03d9ac59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:31:14.408740 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.408715 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kube-api-access-l2z2x" (OuterVolumeSpecName: "kube-api-access-l2z2x") pod "5c1cb364-7dd6-4875-ae28-bcae03d9ac59" (UID: "5c1cb364-7dd6-4875-ae28-bcae03d9ac59"). InnerVolumeSpecName "kube-api-access-l2z2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:31:14.408788 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.408721 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c1cb364-7dd6-4875-ae28-bcae03d9ac59" (UID: "5c1cb364-7dd6-4875-ae28-bcae03d9ac59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:31:14.507688 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.507661 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:14.507688 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.507687 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2z2x\" (UniqueName: \"kubernetes.io/projected/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-kube-api-access-l2z2x\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:14.507832 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.507703 2554 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-isvc-init-fail-b18b30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:14.507832 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.507714 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:14.507832 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:14.507724 2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1cb364-7dd6-4875-ae28-bcae03d9ac59-cabundle-cert\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:31:15.029970 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.029948 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7_5c1cb364-7dd6-4875-ae28-bcae03d9ac59/storage-initializer/1.log" May 25 13:31:15.030291 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.030278 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7_5c1cb364-7dd6-4875-ae28-bcae03d9ac59/storage-initializer/0.log" May 25 13:31:15.030356 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.030311 2554 generic.go:358] "Generic (PLEG): container finished" podID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerID="853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb" exitCode=1 May 25 13:31:15.030410 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.030395 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" May 25 13:31:15.030493 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.030394 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" event={"ID":"5c1cb364-7dd6-4875-ae28-bcae03d9ac59","Type":"ContainerDied","Data":"853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb"} May 25 13:31:15.030493 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.030482 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7" event={"ID":"5c1cb364-7dd6-4875-ae28-bcae03d9ac59","Type":"ContainerDied","Data":"48cdea0e34e5f19f685216f7e3873315b6f3bcd2d27e9e2fa46c968a97bea884"} May 25 13:31:15.030564 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.030501 2554 scope.go:117] "RemoveContainer" containerID="853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb" May 25 13:31:15.038515 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.038500 2554 scope.go:117] "RemoveContainer" containerID="9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4" May 25 13:31:15.045361 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.045340 2554 scope.go:117] "RemoveContainer" containerID="853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb" May 25 13:31:15.045618 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:15.045600 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb\": container with ID starting with 853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb not found: ID does not exist" containerID="853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb" May 25 13:31:15.045673 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.045625 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb"} err="failed to get container status \"853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb\": rpc error: code = NotFound desc = could not find container \"853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb\": container with ID starting with 853302009f3be87ef6c369cbdb7d8237834c24b4c1a2db88f96f807989c1c0bb not found: ID does not exist" May 25 13:31:15.045673 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.045642 2554 scope.go:117] "RemoveContainer" containerID="9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4" May 25 13:31:15.045858 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:31:15.045842 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4\": container with ID starting with 9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4 not found: ID does not exist" containerID="9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4" May 25 13:31:15.045902 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.045863 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4"} err="failed to get container status \"9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4\": rpc error: code = NotFound desc = could not find container \"9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4\": container with ID starting with 9777e485336c565b0d9e2631ec95c77ddbb117f142bfa137204c8f0b4be320e4 not found: ID does not exist" May 25 13:31:15.071164 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.071139 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7"] May 25 13:31:15.075375 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:15.075357 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b18b30-predictor-7986dfbf95-hv9c7"] May 25 13:31:16.902724 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:16.902685 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" path="/var/lib/kubelet/pods/5c1cb364-7dd6-4875-ae28-bcae03d9ac59/volumes" May 25 13:31:17.040106 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:17.040074 2554 generic.go:358] "Generic (PLEG): container finished" podID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerID="7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7" exitCode=0 May 25 13:31:17.040258 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:17.040147 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerDied","Data":"7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7"} May 25 13:31:18.045391 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:18.045356 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerStarted","Data":"17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7"} May 25 13:31:18.045785 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:18.045412 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerStarted","Data":"d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5"} May 25 13:31:18.045785 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:18.045633 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:18.070284 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:18.070238 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podStartSLOduration=7.070221142 podStartE2EDuration="7.070221142s" podCreationTimestamp="2026-05-25 13:31:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:31:18.067921094 +0000 UTC m=+1489.746218707" watchObservedRunningTime="2026-05-25 13:31:18.070221142 +0000 UTC m=+1489.748518745" May 25 13:31:19.048687 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:19.048647 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:19.049817 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:19.049789 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:31:20.051936 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:20.051894 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:31:25.056288 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:25.056259 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:31:25.056862 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:25.056833 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:31:28.896852 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:28.896826 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:31:28.901519 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:28.901492 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:31:35.057194 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:35.057154 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:31:45.057182 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:45.057105 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:31:55.057089 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:31:55.057046 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:32:05.057168 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:05.057127 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:32:15.056872 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:15.056829 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:32:25.058096 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:25.058066 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:32:31.352139 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.352108 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs"] May 25 13:32:31.352618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.352510 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" containerID="cri-o://d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5" gracePeriod=30 May 25 13:32:31.352730 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.352673 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kube-rbac-proxy" containerID="cri-o://17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7" gracePeriod=30 May 25 13:32:31.435673 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.435648 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw"] May 25 13:32:31.435989 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.435977 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerName="storage-initializer" May 25 13:32:31.436037 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.435990 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerName="storage-initializer" May 25 13:32:31.436037 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.436010 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerName="storage-initializer" May 25 13:32:31.436037 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.436015 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerName="storage-initializer" May 25 13:32:31.436129 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.436068 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerName="storage-initializer" May 25 13:32:31.436129 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.436075 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c1cb364-7dd6-4875-ae28-bcae03d9ac59" containerName="storage-initializer" May 25 13:32:31.439347 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.439329 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.441794 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.441773 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\"" May 25 13:32:31.441794 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.441780 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-b5994-predictor-serving-cert\"" May 25 13:32:31.449567 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.449546 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw"] May 25 13:32:31.586151 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.586128 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f889b9f-6acc-4771-8cdf-d11b11396fc9-raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.586248 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.586173 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kserve-provision-location\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.586291 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.586266 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f889b9f-6acc-4771-8cdf-d11b11396fc9-proxy-tls\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.586291 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.586283 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppx7\" (UniqueName: \"kubernetes.io/projected/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kube-api-access-mppx7\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.686874 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.686821 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f889b9f-6acc-4771-8cdf-d11b11396fc9-proxy-tls\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.686874 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.686857 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mppx7\" (UniqueName: \"kubernetes.io/projected/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kube-api-access-mppx7\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.687024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.686884 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f889b9f-6acc-4771-8cdf-d11b11396fc9-raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.687024 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.686917 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kserve-provision-location\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.687319 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.687293 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kserve-provision-location\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.687612 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.687592 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f889b9f-6acc-4771-8cdf-d11b11396fc9-raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.689144 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.689127 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f889b9f-6acc-4771-8cdf-d11b11396fc9-proxy-tls\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.695576 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.695549 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppx7\" (UniqueName: \"kubernetes.io/projected/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kube-api-access-mppx7\") pod \"raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.749921 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.749891 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:31.877169 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:31.877140 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw"] May 25 13:32:31.879786 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:32:31.879755 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f889b9f_6acc_4771_8cdf_d11b11396fc9.slice/crio-74bcd33fcbb8dec48e475314d4d5b597511a892455dc7fc65460f63e85aa1545 WatchSource:0}: Error finding container 74bcd33fcbb8dec48e475314d4d5b597511a892455dc7fc65460f63e85aa1545: Status 404 returned error can't find the container with id 74bcd33fcbb8dec48e475314d4d5b597511a892455dc7fc65460f63e85aa1545 May 25 13:32:32.291546 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:32.291505 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerStarted","Data":"136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c"} May 25 13:32:32.291759 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:32.291551 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerStarted","Data":"74bcd33fcbb8dec48e475314d4d5b597511a892455dc7fc65460f63e85aa1545"} May 25 13:32:32.293367 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:32.293343 2554 generic.go:358] "Generic (PLEG): container finished" podID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerID="17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7" exitCode=2 May 25 13:32:32.293477 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:32.293372 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerDied","Data":"17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7"} May 25 13:32:35.052523 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.052485 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused" May 25 13:32:35.056764 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.056740 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused" May 25 13:32:35.297557 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.297535 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:32:35.305593 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.305541 2554 generic.go:358] "Generic (PLEG): container finished" podID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerID="d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5" exitCode=0 May 25 13:32:35.305688 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.305622 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" May 25 13:32:35.305725 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.305621 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerDied","Data":"d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5"} May 25 13:32:35.305761 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.305730 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs" event={"ID":"ae0250f8-e52a-433c-a0df-6a914c07e576","Type":"ContainerDied","Data":"355353c1d3d03551140e6dc2eb9b604057d995ad0dd3dab72068654cffdd6697"} May 25 13:32:35.305761 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.305752 2554 scope.go:117] "RemoveContainer" containerID="17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7" May 25 13:32:35.315601 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.315576 2554 scope.go:117] "RemoveContainer" containerID="d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5" May 25 13:32:35.324300 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.324275 2554 scope.go:117] "RemoveContainer" containerID="7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7" May 25 13:32:35.331133 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.331116 2554 scope.go:117] "RemoveContainer" containerID="17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7" May 25 13:32:35.331380 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:32:35.331358 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7\": container with ID starting with 17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7 not found: ID does not exist" containerID="17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7" May 25 13:32:35.331508 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.331388 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7"} err="failed to get container status \"17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7\": rpc error: code = NotFound desc = could not find container \"17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7\": container with ID starting with 17774cd3cd8f4ae3db04b71eeacbb25360cd2afb8913f10282489f3a937505d7 not found: ID does not exist" May 25 13:32:35.331508 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.331404 2554 scope.go:117] "RemoveContainer" containerID="d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5" May 25 13:32:35.331787 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:32:35.331768 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5\": container with ID starting with d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5 not found: ID does not exist" containerID="d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5" May 25 13:32:35.331851 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.331796 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5"} err="failed to get container status \"d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5\": rpc error: code = NotFound desc = could not find container \"d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5\": container with ID starting with d282a82cd5e578338e301c904ada632bf1dc997fd2655207a35bda6b0bfff1d5 not found: ID does not exist" May 25 13:32:35.331851 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.331812 2554 scope.go:117] "RemoveContainer" containerID="7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7" May 25 13:32:35.332015 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:32:35.331999 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7\": container with ID starting with 7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7 not found: ID does not exist" containerID="7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7" May 25 13:32:35.332070 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.332019 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7"} err="failed to get container status \"7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7\": rpc error: code = NotFound desc = could not find container \"7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7\": container with ID starting with 7c809465fbe65e24f127932e2e04d123827cab28e58a2bf13c62558e4fd838d7 not found: ID does not exist" May 25 13:32:35.416500 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.416478 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcsz4\" (UniqueName: \"kubernetes.io/projected/ae0250f8-e52a-433c-a0df-6a914c07e576-kube-api-access-kcsz4\") pod \"ae0250f8-e52a-433c-a0df-6a914c07e576\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " May 25 13:32:35.416581 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.416543 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0250f8-e52a-433c-a0df-6a914c07e576-kserve-provision-location\") pod \"ae0250f8-e52a-433c-a0df-6a914c07e576\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " May 25 13:32:35.416581 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.416575 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls\") pod \"ae0250f8-e52a-433c-a0df-6a914c07e576\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " May 25 13:32:35.416676 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.416591 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-a52ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae0250f8-e52a-433c-a0df-6a914c07e576-raw-sklearn-a52ab-kube-rbac-proxy-sar-config\") pod \"ae0250f8-e52a-433c-a0df-6a914c07e576\" (UID: \"ae0250f8-e52a-433c-a0df-6a914c07e576\") " May 25 13:32:35.416912 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.416880 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0250f8-e52a-433c-a0df-6a914c07e576-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae0250f8-e52a-433c-a0df-6a914c07e576" (UID: "ae0250f8-e52a-433c-a0df-6a914c07e576"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:32:35.417002 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.416979 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0250f8-e52a-433c-a0df-6a914c07e576-raw-sklearn-a52ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-a52ab-kube-rbac-proxy-sar-config") pod "ae0250f8-e52a-433c-a0df-6a914c07e576" (UID: "ae0250f8-e52a-433c-a0df-6a914c07e576"). InnerVolumeSpecName "raw-sklearn-a52ab-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:32:35.418414 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.418391 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0250f8-e52a-433c-a0df-6a914c07e576-kube-api-access-kcsz4" (OuterVolumeSpecName: "kube-api-access-kcsz4") pod "ae0250f8-e52a-433c-a0df-6a914c07e576" (UID: "ae0250f8-e52a-433c-a0df-6a914c07e576"). InnerVolumeSpecName "kube-api-access-kcsz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:32:35.418503 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.418447 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae0250f8-e52a-433c-a0df-6a914c07e576" (UID: "ae0250f8-e52a-433c-a0df-6a914c07e576"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:32:35.517506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.517482 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0250f8-e52a-433c-a0df-6a914c07e576-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:32:35.517506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.517504 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae0250f8-e52a-433c-a0df-6a914c07e576-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:32:35.517633 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.517515 2554 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-a52ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae0250f8-e52a-433c-a0df-6a914c07e576-raw-sklearn-a52ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:32:35.517633 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.517525 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcsz4\" (UniqueName: \"kubernetes.io/projected/ae0250f8-e52a-433c-a0df-6a914c07e576-kube-api-access-kcsz4\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:32:35.628293 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.628267 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs"] May 25 13:32:35.632837 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:35.632812 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-a52ab-predictor-6f7fd64669-zzcvs"] May 25 13:32:36.311261 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:36.311229 2554 generic.go:358] "Generic (PLEG): container finished" podID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerID="136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c" exitCode=0 May 25 13:32:36.311615 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:36.311272 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerDied","Data":"136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c"} May 25 13:32:36.901157 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:36.901120 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" path="/var/lib/kubelet/pods/ae0250f8-e52a-433c-a0df-6a914c07e576/volumes" May 25 13:32:37.316898 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:37.316868 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerStarted","Data":"540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654"} May 25 13:32:37.317240 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:37.316907 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerStarted","Data":"ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2"} May 25 13:32:37.317240 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:37.317111 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:37.338117 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:37.338062 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podStartSLOduration=6.338046452 podStartE2EDuration="6.338046452s" podCreationTimestamp="2026-05-25 13:32:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:32:37.336191738 +0000 UTC m=+1569.014489339" watchObservedRunningTime="2026-05-25 13:32:37.338046452 +0000 UTC m=+1569.016344055" May 25 13:32:38.320173 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:38.320136 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:38.321477 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:38.321450 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:32:39.323509 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:39.323470 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:32:44.327916 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:44.327886 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:32:44.328494 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:44.328464 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:32:54.328817 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:32:54.328773 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:33:04.329332 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:04.329292 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:33:14.328482 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:14.328397 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:33:24.329352 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:24.329311 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:33:34.329064 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:34.329026 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:33:44.329027 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:44.328998 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:33:51.526867 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:51.526831 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw"] May 25 13:33:51.527444 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:51.527254 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" containerID="cri-o://ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2" gracePeriod=30 May 25 13:33:51.527444 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:51.527335 2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kube-rbac-proxy" containerID="cri-o://540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654" gracePeriod=30 May 25 13:33:52.579238 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:52.579207 2554 generic.go:358] "Generic (PLEG): container finished" podID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerID="540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654" exitCode=2 May 25 13:33:52.579625 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:52.579287 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerDied","Data":"540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654"} May 25 13:33:54.323739 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:54.323685 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused" May 25 13:33:54.329020 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:54.328987 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused" May 25 13:33:55.275619 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.275589 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:33:55.415776 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.415712 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f889b9f-6acc-4771-8cdf-d11b11396fc9-raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\") pod \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " May 25 13:33:55.416142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.415815 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kserve-provision-location\") pod \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " May 25 13:33:55.416142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.415847 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mppx7\" (UniqueName: \"kubernetes.io/projected/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kube-api-access-mppx7\") pod \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " May 25 13:33:55.416142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.415877 2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f889b9f-6acc-4771-8cdf-d11b11396fc9-proxy-tls\") pod \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\" (UID: \"5f889b9f-6acc-4771-8cdf-d11b11396fc9\") " May 25 13:33:55.416142 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.416096 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f889b9f-6acc-4771-8cdf-d11b11396fc9-raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config") pod "5f889b9f-6acc-4771-8cdf-d11b11396fc9" (UID: "5f889b9f-6acc-4771-8cdf-d11b11396fc9"). InnerVolumeSpecName "raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 25 13:33:55.416314 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.416148 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f889b9f-6acc-4771-8cdf-d11b11396fc9" (UID: "5f889b9f-6acc-4771-8cdf-d11b11396fc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 25 13:33:55.417835 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.417810 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f889b9f-6acc-4771-8cdf-d11b11396fc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f889b9f-6acc-4771-8cdf-d11b11396fc9" (UID: "5f889b9f-6acc-4771-8cdf-d11b11396fc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 25 13:33:55.417966 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.417945 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kube-api-access-mppx7" (OuterVolumeSpecName: "kube-api-access-mppx7") pod "5f889b9f-6acc-4771-8cdf-d11b11396fc9" (UID: "5f889b9f-6acc-4771-8cdf-d11b11396fc9"). InnerVolumeSpecName "kube-api-access-mppx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 25 13:33:55.516642 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.516617 2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kserve-provision-location\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:33:55.516642 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.516640 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mppx7\" (UniqueName: \"kubernetes.io/projected/5f889b9f-6acc-4771-8cdf-d11b11396fc9-kube-api-access-mppx7\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:33:55.516774 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.516652 2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f889b9f-6acc-4771-8cdf-d11b11396fc9-proxy-tls\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:33:55.516774 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.516664 2554 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f889b9f-6acc-4771-8cdf-d11b11396fc9-raw-sklearn-runtime-b5994-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-64.ec2.internal\" DevicePath \"\"" May 25 13:33:55.591776 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.591745 2554 generic.go:358] "Generic (PLEG): container finished" podID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerID="ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2" exitCode=0 May 25 13:33:55.591861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.591783 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerDied","Data":"ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2"} May 25 13:33:55.591861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.591806 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" event={"ID":"5f889b9f-6acc-4771-8cdf-d11b11396fc9","Type":"ContainerDied","Data":"74bcd33fcbb8dec48e475314d4d5b597511a892455dc7fc65460f63e85aa1545"} May 25 13:33:55.591861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.591823 2554 scope.go:117] "RemoveContainer" containerID="540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654" May 25 13:33:55.591861 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.591826 2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw" May 25 13:33:55.603362 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.603341 2554 scope.go:117] "RemoveContainer" containerID="ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2" May 25 13:33:55.610657 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.610641 2554 scope.go:117] "RemoveContainer" containerID="136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c" May 25 13:33:55.615812 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.615790 2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw"] May 25 13:33:55.618501 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.618482 2554 scope.go:117] "RemoveContainer" containerID="540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654" May 25 13:33:55.618775 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:33:55.618751 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654\": container with ID starting with 540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654 not found: ID does not exist" containerID="540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654" May 25 13:33:55.618853 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.618786 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654"} err="failed to get container status \"540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654\": rpc error: code = NotFound desc = could not find container \"540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654\": container with ID starting with 540c4d5017bc2588d046984fde945631e87f95c5dfbb2f3ac37c7c5afd35f654 not found: ID does not exist" May 25 13:33:55.618853 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.618811 2554 scope.go:117] "RemoveContainer" containerID="ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2" May 25 13:33:55.619117 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:33:55.619097 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2\": container with ID starting with ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2 not found: ID does not exist" containerID="ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2" May 25 13:33:55.619189 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.619122 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2"} err="failed to get container status \"ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2\": rpc error: code = NotFound desc = could not find container \"ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2\": container with ID starting with ae9645a14af1a3a1bba5cc6f461f0009c0fc0466e1a4c899e9ef173be82293b2 not found: ID does not exist" May 25 13:33:55.619189 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.619142 2554 scope.go:117] "RemoveContainer" containerID="136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c" May 25 13:33:55.619408 ip-10-0-135-64 kubenswrapper[2554]: E0525 13:33:55.619389 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c\": container with ID starting with 136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c not found: ID does not exist" containerID="136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c" May 25 13:33:55.619491 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.619417 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c"} err="failed to get container status \"136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c\": rpc error: code = NotFound desc = could not find container \"136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c\": container with ID starting with 136311081fa8f155fd1339563b21b8bafd827aa45953cc0813b276d6d93fac4c not found: ID does not exist" May 25 13:33:55.619969 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:55.619953 2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-b5994-predictor-65f684bd7f-2dztw"] May 25 13:33:56.901865 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:33:56.901832 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" path="/var/lib/kubelet/pods/5f889b9f-6acc-4771-8cdf-d11b11396fc9/volumes" May 25 13:34:19.006544 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.006504 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kt9b5/must-gather-hcg72"] May 25 13:34:19.007029 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007005 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kube-rbac-proxy" May 25 13:34:19.007100 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007035 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kube-rbac-proxy" May 25 13:34:19.007100 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007060 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="storage-initializer" May 25 13:34:19.007100 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007070 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="storage-initializer" May 25 13:34:19.007100 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007084 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" May 25 13:34:19.007100 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007093 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007114 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kube-rbac-proxy" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007125 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kube-rbac-proxy" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007137 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007145 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007158 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="storage-initializer" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007167 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="storage-initializer" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007256 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kube-rbac-proxy" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007267 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae0250f8-e52a-433c-a0df-6a914c07e576" containerName="kserve-container" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007279 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kube-rbac-proxy" May 25 13:34:19.007363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.007292 2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f889b9f-6acc-4771-8cdf-d11b11396fc9" containerName="kserve-container" May 25 13:34:19.010763 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.010736 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.013171 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.013149 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt9b5\"/\"kube-root-ca.crt\"" May 25 13:34:19.013912 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.013891 2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kt9b5\"/\"default-dockercfg-hkc44\"" May 25 13:34:19.014029 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.013924 2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt9b5\"/\"openshift-service-ca.crt\"" May 25 13:34:19.017786 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.017764 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt9b5/must-gather-hcg72"] May 25 13:34:19.099525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.099491 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f75a2de7-0d1f-4e44-8c94-f461c55c58ca-must-gather-output\") pod \"must-gather-hcg72\" (UID: \"f75a2de7-0d1f-4e44-8c94-f461c55c58ca\") " pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.099618 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.099548 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdvf\" (UniqueName: \"kubernetes.io/projected/f75a2de7-0d1f-4e44-8c94-f461c55c58ca-kube-api-access-4zdvf\") pod \"must-gather-hcg72\" (UID: \"f75a2de7-0d1f-4e44-8c94-f461c55c58ca\") " pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.200572 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.200545 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f75a2de7-0d1f-4e44-8c94-f461c55c58ca-must-gather-output\") pod \"must-gather-hcg72\" (UID: \"f75a2de7-0d1f-4e44-8c94-f461c55c58ca\") " pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.200686 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.200588 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdvf\" (UniqueName: \"kubernetes.io/projected/f75a2de7-0d1f-4e44-8c94-f461c55c58ca-kube-api-access-4zdvf\") pod \"must-gather-hcg72\" (UID: \"f75a2de7-0d1f-4e44-8c94-f461c55c58ca\") " pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.200885 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.200866 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f75a2de7-0d1f-4e44-8c94-f461c55c58ca-must-gather-output\") pod \"must-gather-hcg72\" (UID: \"f75a2de7-0d1f-4e44-8c94-f461c55c58ca\") " pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.208454 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.208416 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdvf\" (UniqueName: \"kubernetes.io/projected/f75a2de7-0d1f-4e44-8c94-f461c55c58ca-kube-api-access-4zdvf\") pod \"must-gather-hcg72\" (UID: \"f75a2de7-0d1f-4e44-8c94-f461c55c58ca\") " pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.334798 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.334777 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt9b5/must-gather-hcg72" May 25 13:34:19.462378 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.462353 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt9b5/must-gather-hcg72"] May 25 13:34:19.464257 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:34:19.464230 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf75a2de7_0d1f_4e44_8c94_f461c55c58ca.slice/crio-5ea7ab5f099963c234d4bd55161d3f55c0c2ecba9ab1359196aa7dab66175f5e WatchSource:0}: Error finding container 5ea7ab5f099963c234d4bd55161d3f55c0c2ecba9ab1359196aa7dab66175f5e: Status 404 returned error can't find the container with id 5ea7ab5f099963c234d4bd55161d3f55c0c2ecba9ab1359196aa7dab66175f5e May 25 13:34:19.678248 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:19.678185 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt9b5/must-gather-hcg72" event={"ID":"f75a2de7-0d1f-4e44-8c94-f461c55c58ca","Type":"ContainerStarted","Data":"5ea7ab5f099963c234d4bd55161d3f55c0c2ecba9ab1359196aa7dab66175f5e"} May 25 13:34:20.683649 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:20.683610 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt9b5/must-gather-hcg72" event={"ID":"f75a2de7-0d1f-4e44-8c94-f461c55c58ca","Type":"ContainerStarted","Data":"f7860f36e45c4da5fe5b5d3b80dbccb63bd7e0ee11610c50af48f1abacacf135"} May 25 13:34:20.684099 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:20.683657 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt9b5/must-gather-hcg72" event={"ID":"f75a2de7-0d1f-4e44-8c94-f461c55c58ca","Type":"ContainerStarted","Data":"13853121265c601b422d4979a0a76bce20c9ccab4e354a14870fce69e328b2ba"} May 25 13:34:20.700164 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:20.700110 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kt9b5/must-gather-hcg72" podStartSLOduration=1.8857056509999999 podStartE2EDuration="2.700093823s" podCreationTimestamp="2026-05-25 13:34:18 +0000 UTC" firstStartedPulling="2026-05-25 13:34:19.466141532 +0000 UTC m=+1671.144439116" lastFinishedPulling="2026-05-25 13:34:20.280529707 +0000 UTC m=+1671.958827288" observedRunningTime="2026-05-25 13:34:20.698520769 +0000 UTC m=+1672.376818383" watchObservedRunningTime="2026-05-25 13:34:20.700093823 +0000 UTC m=+1672.378391428" May 25 13:34:21.705743 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:21.705714 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-khdhn_db21c73e-1377-495c-b026-629cacfc1a70/global-pull-secret-syncer/0.log" May 25 13:34:21.869757 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:21.869723 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sq2c5_c8bac7fa-29df-4ca5-8451-ad222a9d671c/konnectivity-agent/0.log" May 25 13:34:21.914071 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:21.914041 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-64.ec2.internal_48fb10cd0fb028145d43ec292fc620af/haproxy/0.log" May 25 13:34:25.266636 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.266606 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0113e92-ebec-4a04-90c6-68078134db4b/alertmanager/0.log" May 25 13:34:25.289215 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.289180 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0113e92-ebec-4a04-90c6-68078134db4b/config-reloader/0.log" May 25 13:34:25.313363 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.313334 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0113e92-ebec-4a04-90c6-68078134db4b/kube-rbac-proxy-web/0.log" May 25 13:34:25.339352 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.339314 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0113e92-ebec-4a04-90c6-68078134db4b/kube-rbac-proxy/0.log" May 25 13:34:25.361671 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.361612 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0113e92-ebec-4a04-90c6-68078134db4b/kube-rbac-proxy-metric/0.log" May 25 13:34:25.385013 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.384986 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0113e92-ebec-4a04-90c6-68078134db4b/prom-label-proxy/0.log" May 25 13:34:25.409387 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.409355 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0113e92-ebec-4a04-90c6-68078134db4b/init-config-reloader/0.log" May 25 13:34:25.447727 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.447694 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-zb92d_6aeb4909-83b4-4b8a-ac96-d20a74970d44/cluster-monitoring-operator/0.log" May 25 13:34:25.576073 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.576035 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-cnrrg_f235c12c-ecc9-4199-9343-bdeb4dc50486/monitoring-plugin/0.log" May 25 13:34:25.685382 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.685349 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w72sz_08dc7950-719f-492b-8afe-e453b596a78d/node-exporter/0.log" May 25 13:34:25.706506 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.706477 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w72sz_08dc7950-719f-492b-8afe-e453b596a78d/kube-rbac-proxy/0.log" May 25 13:34:25.730345 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:25.730317 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w72sz_08dc7950-719f-492b-8afe-e453b596a78d/init-textfile/0.log" May 25 13:34:26.140015 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:26.139985 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-p8g64_188626a6-0721-460f-9937-a93d4413732f/prometheus-operator-admission-webhook/0.log" May 25 13:34:26.169176 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:26.169117 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f9b66f-svp5b_c0b3f96f-81f4-4063-a627-069319cc24a1/telemeter-client/0.log" May 25 13:34:26.196412 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:26.196379 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f9b66f-svp5b_c0b3f96f-81f4-4063-a627-069319cc24a1/reload/0.log" May 25 13:34:26.226030 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:26.226003 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f9b66f-svp5b_c0b3f96f-81f4-4063-a627-069319cc24a1/kube-rbac-proxy/0.log" May 25 13:34:27.953041 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:27.952963 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/2.log" May 25 13:34:27.958168 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:27.958147 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-jfbcj_077160a2-2714-4cf9-961e-4169dc361eaa/console-operator/3.log" May 25 13:34:28.307542 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.307504 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74bb8fbbdc-vk76q_3997723e-5f8f-4631-b047-89eeb7b79978/console/0.log" May 25 13:34:28.337283 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.337255 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-9sgf6_5f52192a-99b2-4f70-be41-8a4d7037f25f/download-server/0.log" May 25 13:34:28.673225 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.673182 2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr"] May 25 13:34:28.678844 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.678820 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.686389 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.686366 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr"] May 25 13:34:28.694355 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.694322 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjp85\" (UniqueName: \"kubernetes.io/projected/4b8584f5-303f-4430-96d3-b151be126072-kube-api-access-gjp85\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.694505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.694416 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-podres\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.694505 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.694499 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-sys\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.694639 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.694544 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-lib-modules\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.694639 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.694578 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-proc\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.752504 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.752478 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-8ngrf_97a552c2-8f68-4558-85d3-300397fe201b/volume-data-source-validator/0.log" May 25 13:34:28.795315 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795279 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-podres\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795349 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-sys\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795392 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-lib-modules\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795461 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-proc\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795472 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-sys\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795525 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795496 2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gjp85\" (UniqueName: \"kubernetes.io/projected/4b8584f5-303f-4430-96d3-b151be126072-kube-api-access-gjp85\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795476 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-podres\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795534 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-proc\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.795824 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.795587 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8584f5-303f-4430-96d3-b151be126072-lib-modules\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.804530 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.804504 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjp85\" (UniqueName: \"kubernetes.io/projected/4b8584f5-303f-4430-96d3-b151be126072-kube-api-access-gjp85\") pod \"perf-node-gather-daemonset-ng9hr\" (UID: \"4b8584f5-303f-4430-96d3-b151be126072\") " pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:28.989558 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:28.989529 2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:29.144569 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.144354 2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr"] May 25 13:34:29.148106 ip-10-0-135-64 kubenswrapper[2554]: W0525 13:34:29.148066 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b8584f5_303f_4430_96d3_b151be126072.slice/crio-13fbadd46901ec653017334205dfe79b7754fe7af843aec064c575073e099921 WatchSource:0}: Error finding container 13fbadd46901ec653017334205dfe79b7754fe7af843aec064c575073e099921: Status 404 returned error can't find the container with id 13fbadd46901ec653017334205dfe79b7754fe7af843aec064c575073e099921 May 25 13:34:29.151219 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.150319 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 25 13:34:29.475855 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.475820 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h8xqz_a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8/dns/0.log" May 25 13:34:29.497211 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.497188 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h8xqz_a0c708f1-3e3a-49c2-a692-9f1cf2eb36e8/kube-rbac-proxy/0.log" May 25 13:34:29.563962 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.563913 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d5p4k_e460f6bd-9a46-46d2-870f-f4d203cf05fd/dns-node-resolver/0.log" May 25 13:34:29.722775 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.722742 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" event={"ID":"4b8584f5-303f-4430-96d3-b151be126072","Type":"ContainerStarted","Data":"5623cebfd4202b5650a154e7e941ee32495bbe559736d0ce0359b09d724644e3"} May 25 13:34:29.722926 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.722781 2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" event={"ID":"4b8584f5-303f-4430-96d3-b151be126072","Type":"ContainerStarted","Data":"13fbadd46901ec653017334205dfe79b7754fe7af843aec064c575073e099921"} May 25 13:34:29.722926 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.722862 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:29.754753 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:29.754709 2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" podStartSLOduration=1.754698009 podStartE2EDuration="1.754698009s" podCreationTimestamp="2026-05-25 13:34:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:34:29.751362825 +0000 UTC m=+1681.429660427" watchObservedRunningTime="2026-05-25 13:34:29.754698009 +0000 UTC m=+1681.432995646" May 25 13:34:30.093552 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:30.093512 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78f78dfc48-kpjv4_df3bdbc0-4ecb-4a46-9efa-717313373f35/registry/0.log" May 25 13:34:30.113766 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:30.113744 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5brhb_db1bb1ec-d3b1-43f1-b351-de93dee1b1ea/node-ca/0.log" May 25 13:34:31.223692 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:31.223645 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-64d4d_232bc31b-feac-4e2b-9fb3-f1d3c3a0a075/serve-healthcheck-canary/0.log" May 25 13:34:31.618264 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:31.618232 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-vt6mr_69d027ad-c810-423a-98fa-54547bad1795/insights-operator/0.log" May 25 13:34:31.619213 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:31.619191 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-vt6mr_69d027ad-c810-423a-98fa-54547bad1795/insights-operator/1.log" May 25 13:34:31.707088 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:31.707071 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tg88x_17da813c-56ef-431e-906e-02d26698b3ab/kube-rbac-proxy/0.log" May 25 13:34:31.729184 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:31.729166 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tg88x_17da813c-56ef-431e-906e-02d26698b3ab/exporter/0.log" May 25 13:34:31.752361 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:31.752342 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tg88x_17da813c-56ef-431e-906e-02d26698b3ab/extractor/0.log" May 25 13:34:33.841046 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:33.841010 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-vb877_31d2a561-feb3-44cb-a2b7-6f4f69008f75/manager/0.log" May 25 13:34:33.952175 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:33.952139 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-ml4bh_a4b52f83-ae35-4469-b32c-58a0630496cc/manager/0.log" May 25 13:34:35.739066 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:35.739038 2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kt9b5/perf-node-gather-daemonset-ng9hr" May 25 13:34:37.758895 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:37.758867 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-5sqj4_e1670f1d-70e9-440c-93f4-daf299a206d8/migrator/0.log" May 25 13:34:37.779942 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:37.779909 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-5sqj4_e1670f1d-70e9-440c-93f4-daf299a206d8/graceful-termination/0.log" May 25 13:34:38.172807 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:38.172777 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-9669r_720a0198-112a-4272-b799-e51dbaf56832/kube-storage-version-migrator-operator/1.log" May 25 13:34:38.173935 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:38.173910 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-9669r_720a0198-112a-4272-b799-e51dbaf56832/kube-storage-version-migrator-operator/0.log" May 25 13:34:39.216964 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.216926 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7wr8_48876559-9080-4bcd-9ae5-0efdddcd6d7d/kube-multus-additional-cni-plugins/0.log" May 25 13:34:39.241499 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.241473 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7wr8_48876559-9080-4bcd-9ae5-0efdddcd6d7d/egress-router-binary-copy/0.log" May 25 13:34:39.263081 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.263059 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7wr8_48876559-9080-4bcd-9ae5-0efdddcd6d7d/cni-plugins/0.log" May 25 13:34:39.285923 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.285893 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7wr8_48876559-9080-4bcd-9ae5-0efdddcd6d7d/bond-cni-plugin/0.log" May 25 13:34:39.316268 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.316241 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7wr8_48876559-9080-4bcd-9ae5-0efdddcd6d7d/routeoverride-cni/0.log" May 25 13:34:39.342318 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.342292 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7wr8_48876559-9080-4bcd-9ae5-0efdddcd6d7d/whereabouts-cni-bincopy/0.log" May 25 13:34:39.366066 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.366046 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7wr8_48876559-9080-4bcd-9ae5-0efdddcd6d7d/whereabouts-cni/0.log" May 25 13:34:39.803148 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.803100 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mkxpd_3655e2cf-b173-414d-8487-626406b3f39d/kube-multus/0.log" May 25 13:34:39.826834 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.826796 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2228g_1e2b6fca-f227-41c5-9d85-6c3b82019d24/network-metrics-daemon/0.log" May 25 13:34:39.853946 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:39.853929 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2228g_1e2b6fca-f227-41c5-9d85-6c3b82019d24/kube-rbac-proxy/0.log" May 25 13:34:40.960762 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:40.960724 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/ovn-controller/0.log" May 25 13:34:40.989641 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:40.989615 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/ovn-acl-logging/0.log" May 25 13:34:41.017149 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:41.017127 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/kube-rbac-proxy-node/0.log" May 25 13:34:41.043597 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:41.043579 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/kube-rbac-proxy-ovn-metrics/0.log" May 25 13:34:41.063926 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:41.063906 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/northd/0.log" May 25 13:34:41.089639 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:41.089618 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/nbdb/0.log" May 25 13:34:41.111879 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:41.111863 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/sbdb/0.log" May 25 13:34:41.224241 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:41.224185 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wtp2_7492a595-944d-46bb-a1b3-440331bc5331/ovnkube-controller/0.log" May 25 13:34:42.556005 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:42.555974 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bhlgr_48e5ebb0-8e9f-498d-bb9e-33568f8b2b02/network-check-target-container/0.log" May 25 13:34:43.604577 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:43.604550 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-b682t_4f51c159-d778-4d89-a010-f265bc961088/iptables-alerter/0.log" May 25 13:34:44.337468 ip-10-0-135-64 kubenswrapper[2554]: I0525 13:34:44.337401 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bxl5h_cd8ab44b-a4bb-4222-afe1-a46b0b747cfa/tuned/0.log"