May 26 11:22:20.794158 ip-10-0-132-203 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 26 11:22:20.794171 ip-10-0-132-203 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 26 11:22:20.794178 ip-10-0-132-203 systemd[1]: kubelet.service: Failed with result 'resources'.
May 26 11:22:20.794384 ip-10-0-132-203 systemd[1]: Failed to start Kubernetes Kubelet.
May 26 11:22:30.953353 ip-10-0-132-203 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 26 11:22:30.953369 ip-10-0-132-203 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 82e808a57467404ba4ec3d8573cd0305 --
May 26 11:24:38.045373 ip-10-0-132-203 systemd[1]: Starting Kubernetes Kubelet...
May 26 11:24:38.545255 ip-10-0-132-203 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:38.545255 ip-10-0-132-203 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 11:24:38.545255 ip-10-0-132-203 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:38.545255 ip-10-0-132-203 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 11:24:38.545255 ip-10-0-132-203 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:38.548069 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.547947    2557 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 11:24:38.551799 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551785    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:38.551799 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551799    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551802    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551805    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551808    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551811    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551816    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551820    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551823    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551827    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551830    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551833    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551836    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551839    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551848    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551852    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551855    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551857    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551860    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551863    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:38.551858 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551866    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551869    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551872    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551875    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551877    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551880    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551883    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551885    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551888    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551890    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551893    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551895    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551898    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551901    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551903    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551906    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551908    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551911    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551914    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551917    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551919    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:38.552327 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551922    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551925    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551928    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551930    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551932    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551935    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551938    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551941    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551943    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551946    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551948    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551952    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551956    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551959    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551961    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551964    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551966    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551968    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551971    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:38.552827 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551973    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551976    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551979    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551981    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551984    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551987    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551992    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551995    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.551997    2557 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552000    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552003    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552006    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552008    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552010    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552014    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552017    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552019    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552022    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552024    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552027    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:38.553300 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552030    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552033    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552035    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552038    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552041    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.552043    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553086    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553093    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553097    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553100    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553103    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553106    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553109    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553111    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553114    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553117    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553119    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553122    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553125    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553128    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:38.553815 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553131    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553134    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553136    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553139    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553142    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553144    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553147    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553150    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553152    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553155    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553157    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553160    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553163    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553165    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553168    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553171    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553173    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553176    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553179    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553181    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:38.554314 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553184    2557 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553188    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553192    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553195    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553198    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553200    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553203    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553205    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553208    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553211    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553213    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553216    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553219    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553221    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553224    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553226    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553229    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553231    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553234    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:38.554803 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553237    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553240    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553243    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553245    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553248    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553251    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553253    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553256    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553258    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553262    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553266    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553269    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553272    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553275    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553277    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553280    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553283    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553286    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553288    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:38.555276 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553291    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553294    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553296    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553299    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553302    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553305    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553308    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553310    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553313    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553316    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553318    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553321    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553323    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553326    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553391    2557 flags.go:64] FLAG: --address="0.0.0.0"
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553402    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553408    2557 flags.go:64] FLAG: --anonymous-auth="true"
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553413    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553417    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553421    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553425    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 11:24:38.555740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553429    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553432    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553435    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553439    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553442    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553445    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553448    2557 flags.go:64] FLAG: --cgroup-root=""
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553451    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553454    2557 flags.go:64] FLAG: --client-ca-file=""
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553457    2557 flags.go:64] FLAG: --cloud-config=""
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553459    2557 flags.go:64] FLAG: --cloud-provider="external"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553462    2557 flags.go:64] FLAG: --cluster-dns="[]"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553466    2557 flags.go:64] FLAG: --cluster-domain=""
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553469    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553472    2557 flags.go:64] FLAG: --config-dir=""
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553476    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553479    2557 flags.go:64] FLAG: --container-log-max-files="5"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553483    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553486    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553490    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553493    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553496    2557 flags.go:64] FLAG: --contention-profiling="false"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553499    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553502    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553505    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 11:24:38.556274 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553507    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553511    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553514    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553518    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553521    2557 flags.go:64] FLAG: --enable-load-reader="false"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553524    2557 flags.go:64] FLAG: --enable-server="true"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553527    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553531    2557 flags.go:64] FLAG: --event-burst="100"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553534    2557 flags.go:64] FLAG: --event-qps="50"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553537    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553540    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553543    2557 flags.go:64] FLAG: --eviction-hard=""
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553547    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553550    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553553    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553556    2557 flags.go:64] FLAG: --eviction-soft=""
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553559    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553562    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553565    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553568    2557 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553571    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553573    2557 flags.go:64] FLAG: --fail-swap-on="true"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553576    2557 flags.go:64] FLAG: --feature-gates=""
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553580    2557 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553583    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 11:24:38.556880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553586    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553590    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553593    2557 flags.go:64] FLAG: --healthz-port="10248"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553596    2557 flags.go:64] FLAG: --help="false"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553599    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-132-203.ec2.internal"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553602    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553605    2557 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553608    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553611    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553614    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553617    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553620    2557 flags.go:64] FLAG: --image-service-endpoint=""
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553623    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553626    2557 flags.go:64] FLAG: --kube-api-burst="100"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553628    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553631    2557 flags.go:64] FLAG: --kube-api-qps="50"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553634    2557 flags.go:64] FLAG: --kube-reserved=""
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553637    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553640    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553643    2557 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553646    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553649    2557 flags.go:64] FLAG: --lock-file=""
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553652    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553655    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 11:24:38.557495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553658    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553663    2557 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553666    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553669    2557 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553672    2557 flags.go:64] FLAG: --logging-format="text"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553674    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553678    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553681    2557 flags.go:64] FLAG: --manifest-url=""
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553684    2557 flags.go:64] FLAG: --manifest-url-header=""
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553688    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553691    2557 flags.go:64] FLAG: --max-open-files="1000000"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553695    2557 flags.go:64] FLAG: --max-pods="110"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553698    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553701    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553704    2557 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553706    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553709    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553712    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553715    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553722    2557 flags.go:64] FLAG: --node-status-max-images="50"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553725    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553727    2557 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553730    2557 flags.go:64] FLAG: --pod-cidr=""
May 26 11:24:38.558094 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553733    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553738    2557 flags.go:64] FLAG: --pod-manifest-path=""
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553742    2557 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553745    2557 flags.go:64] FLAG: --pods-per-core="0"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553748    2557 flags.go:64] FLAG: --port="10250"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553751    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553753    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e290c2c6a525ee21"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553757    2557 flags.go:64] FLAG: --qos-reserved=""
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553760    2557 flags.go:64] FLAG: --read-only-port="10255"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553762    2557 flags.go:64] FLAG: --register-node="true"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553765    2557 flags.go:64] FLAG: --register-schedulable="true"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553768    2557 flags.go:64] FLAG: --register-with-taints=""
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553771    2557 flags.go:64] FLAG: --registry-burst="10"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553774    2557 flags.go:64] FLAG: --registry-qps="5"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553777    2557 flags.go:64] FLAG: --reserved-cpus=""
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553780    2557 flags.go:64] FLAG: --reserved-memory=""
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553783    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553786    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553789    2557 flags.go:64] FLAG: --rotate-certificates="false"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553792    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553795    2557 flags.go:64] FLAG: --runonce="false"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553798    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553801    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553804    2557 flags.go:64] FLAG: --seccomp-default="false"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553807    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553810    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 11:24:38.558683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553813    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553816    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553819    2557 flags.go:64] FLAG: --storage-driver-password="root"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553822    2557 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553826    2557 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553828    2557 flags.go:64] FLAG: --storage-driver-user="root"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553831    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553834    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553840    2557 flags.go:64] FLAG: --system-cgroups=""
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553843    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553848    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553851    2557 flags.go:64] FLAG: --tls-cert-file=""
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553854    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553858    2557 flags.go:64] FLAG: --tls-min-version=""
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553860    2557 flags.go:64] FLAG: --tls-private-key-file=""
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553863    2557 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553866    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553868    2557 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553871    2557 flags.go:64] FLAG: --v="2"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553875    2557 flags.go:64] FLAG: --version="false"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553879    2557 flags.go:64] FLAG: --vmodule=""
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553883    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.553887    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553986    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:38.559329 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553990    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553994    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.553997    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554000    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554002    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554007    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554010    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554013    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554015    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554018    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554021    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554024    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554026    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554029    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554031    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554034    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554038    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554041    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554043    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:38.559898 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554046    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554048    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554051    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554054    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554056    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554059    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554062    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554064    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554066    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554069    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554086    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554092    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554094    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554097    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554100    2557 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554102    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554105    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554108    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554111    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554113    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:38.560559 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554116    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554119    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554121    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554124    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554126    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554129    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554131    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554134    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554136    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554140    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554143    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554145    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554148    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554150    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554155    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554158    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554161    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554164    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554167    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554170    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:38.561451 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554173    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554176    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554178    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554183    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554185    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554188    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554191    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554194    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554197    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554200    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554203    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554206    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554209    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554211    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554214    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554217    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554219    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554222    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554225    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554227    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:38.562325 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554230    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:38.563059 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554234    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:38.563059 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554236    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:38.563059 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554239    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:38.563059 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554241    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:38.563059 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.554244    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:38.563059 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.554993    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:38.563558 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.563538    2557 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 11:24:38.563621 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.563561    2557 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 11:24:38.563666 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563656    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:38.563666 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563664    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563669    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563674    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563678    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563683    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563687    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563691    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563695    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563700    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563704    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563708    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563712    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563716    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563720    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563725    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563729    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563733    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563737    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563742    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:38.563753 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563746    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563750    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563754    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563759    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563763    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563767    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563771    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563775    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563780    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563784    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563788    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563801    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563805    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563809    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563813    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563820    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563825    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563829    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563833    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:38.564654 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563838    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563844    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563848    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563853    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563857    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563861    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563865    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563869    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563874    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563878    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563882    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563886    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563890    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563895    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563899    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563904    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563908    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563912    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563916    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563921    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:38.565237 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563925    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563929    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563933    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563938    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563942    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563954    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563959    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563963    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563967    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563972    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563976    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563980    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563984    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563989    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563993    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.563998    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564002    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564006    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564010    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564015    2557 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:38.565784 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564019    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564023    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564027    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564031    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564035    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564039    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564043    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.564052    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564285    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564294    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564300    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564304    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564309    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564313    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564318    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564322    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:38.566352 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564326    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564330    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564345    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564349    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564353    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564358    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564362    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564367    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564371    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564375    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564379    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564383    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564389    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564396    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564401    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564405    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564409    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564416    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564421    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:38.567022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564426    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564431    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564435    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564438    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564443    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564447    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564451    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564455    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564460    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564464    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564468    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564473    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564477    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564481    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564485    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564489    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564501    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564505    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564509    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:38.567837 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564514    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564518    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564522    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564526    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564530    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564534    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564538    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564543    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564547    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564551    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564555    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564559    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564564    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564568    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564572    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564575    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564579    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564584    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564588    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564592    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:38.568380 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564596    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564600    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564604    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564609    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564613    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564617    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564621    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564625    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564629    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564633    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564644    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564648    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564653    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564657    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564661    2557 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564665    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564669    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564673    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564678    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:38.568988 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:38.564682    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:38.569466 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.564689    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:38.569466 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.565553    2557 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 11:24:38.569466 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.568548    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 11:24:38.569689 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.569676    2557 server.go:1019] "Starting client certificate rotation"
May 26 11:24:38.569813 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.569795    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:38.569850 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.569831    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:38.603392 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.603374    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:38.605124 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.605107    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:38.620462 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.620448    2557 log.go:25] "Validated CRI v1 runtime API"
May 26 11:24:38.626244 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.626226    2557 log.go:25] "Validated CRI v1 image API"
May 26 11:24:38.627667 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.627647    2557 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 11:24:38.633271 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.633254    2557 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:38.635540 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.635520    2557 fs.go:135] Filesystem UUIDs: map[32c48f9e-fa57-48f5-8d33-84706decbf50:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b4cc7f81-236c-4e1e-af58-85af34a08bf9:/dev/nvme0n1p3]
May 26 11:24:38.635622 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.635540    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 11:24:38.639742 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.639630    2557 manager.go:217] Machine: {Timestamp:2026-05-26 11:24:38.639150944 +0000 UTC m=+0.461535160 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089562 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2524abdeb67d2448ec111d51cd0685 SystemUUID:ec2524ab-deb6-7d24-48ec-111d51cd0685 BootID:82e808a5-7467-404b-a4ec-3d8573cd0305 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cc:b2:bd:8e:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cc:b2:bd:8e:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:8a:fa:d7:1c:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 11:24:38.640862 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.640850    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 11:24:38.640959 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.640943    2557 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 11:24:38.646432 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.646401    2557 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 11:24:38.646619 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.646436    2557 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-203.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 11:24:38.646698 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.646634    2557 topology_manager.go:138] "Creating topology manager with none policy"
May 26 11:24:38.646698 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.646648    2557 container_manager_linux.go:306] "Creating device plugin manager"
May 26 11:24:38.646698 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.646658    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:38.647694 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.647682    2557 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:38.649553 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.649541    2557 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:38.649674 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.649663    2557 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 11:24:38.652469 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.652458    2557 kubelet.go:491] "Attempting to sync node with API server"
May 26 11:24:38.652537 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.652475    2557 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 11:24:38.652537 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.652492    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 11:24:38.652537 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.652504    2557 kubelet.go:397] "Adding apiserver pod source"
May 26 11:24:38.652537 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.652515    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 11:24:38.653911 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.653891    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:38.653985 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.653917    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:38.657395 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.657381    2557 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 11:24:38.659559 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.659546    2557 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 11:24:38.660437 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660427    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 11:24:38.660481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660443    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 11:24:38.660481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660449    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 11:24:38.660481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660454    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 11:24:38.660481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660462    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 11:24:38.660481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660468    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 11:24:38.660481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660473    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 11:24:38.660481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660479    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 11:24:38.660665 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660486    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 11:24:38.660665 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660491    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 11:24:38.660665 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660504    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 11:24:38.660957 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.660949    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 11:24:38.662715 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.662702    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 11:24:38.662748 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.662718    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 11:24:38.665171 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.665152    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 11:24:38.665217 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.665155    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 11:24:38.665618 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.665606    2557 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 11:24:38.665650 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.665645    2557 server.go:1295] "Started kubelet"
May 26 11:24:38.665729 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.665688    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 11:24:38.665799 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.665753    2557 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 11:24:38.665854 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.665843    2557 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 11:24:38.666353 ip-10-0-132-203 systemd[1]: Started Kubernetes Kubelet.
May 26 11:24:38.667290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.667273    2557 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 11:24:38.668187 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.668174    2557 server.go:317] "Adding debug handlers to kubelet server"
May 26 11:24:38.674638 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.674619    2557 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-203.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 11:24:38.675666 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.674530    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-203.ec2.internal.18b319d49afea809  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-203.ec2.internal,UID:ip-10-0-132-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-203.ec2.internal,},FirstTimestamp:2026-05-26 11:24:38.665619465 +0000 UTC m=+0.488003681,LastTimestamp:2026-05-26 11:24:38.665619465 +0000 UTC m=+0.488003681,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-203.ec2.internal,}"
May 26 11:24:38.676354 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.676336    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 11:24:38.677029 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.676982    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 11:24:38.677162 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.677110    2557 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 11:24:38.677162 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.677120    2557 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 11:24:38.677258 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.677218    2557 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 11:24:38.677258 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.677255    2557 reconstruct.go:97] "Volume reconstruction finished"
May 26 11:24:38.677341 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.677263    2557 reconciler.go:26] "Reconciler: start to sync state"
May 26 11:24:38.677559 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.677544    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cnh5j"
May 26 11:24:38.678180 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.678162    2557 factory.go:55] Registering systemd factory
May 26 11:24:38.678256 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.678199    2557 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 11:24:38.678256 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.678222    2557 factory.go:223] Registration of the systemd container factory successfully
May 26 11:24:38.678532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.678431    2557 factory.go:153] Registering CRI-O factory
May 26 11:24:38.678532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.678444    2557 factory.go:223] Registration of the crio container factory successfully
May 26 11:24:38.678532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.678491    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 11:24:38.678532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.678508    2557 factory.go:103] Registering Raw factory
May 26 11:24:38.678532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.678524    2557 manager.go:1196] Started watching for new ooms in manager
May 26 11:24:38.678866 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.678835    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:38.679042 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.679013    2557 manager.go:319] Starting recovery of all containers
May 26 11:24:38.680800 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.680772    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 26 11:24:38.680899 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.680865    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 26 11:24:38.686573 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.686552    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cnh5j"
May 26 11:24:38.691389 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.691264    2557 manager.go:324] Recovery completed
May 26 11:24:38.695521 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.695508    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:38.697182 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.697165    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:38.697257 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.697198    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:38.697257 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.697212    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:38.697712 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.697698    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 11:24:38.697712 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.697711    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 11:24:38.697798 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.697724    2557 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:38.699476 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.699402    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-203.ec2.internal.18b319d49ce0432d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-203.ec2.internal,UID:ip-10-0-132-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-203.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-203.ec2.internal,},FirstTimestamp:2026-05-26 11:24:38.697181997 +0000 UTC m=+0.519566215,LastTimestamp:2026-05-26 11:24:38.697181997 +0000 UTC m=+0.519566215,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-203.ec2.internal,}"
May 26 11:24:38.701360 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.701347    2557 policy_none.go:49] "None policy: Start"
May 26 11:24:38.701421 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.701363    2557 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 11:24:38.701421 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.701373    2557 state_mem.go:35] "Initializing new in-memory state store"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.745288    2557 manager.go:341] "Starting Device Plugin manager"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.745318    2557 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.745327    2557 server.go:85] "Starting device plugin registration server"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.745529    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.745540    2557 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.745643    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.745714    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.745723    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.746253    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 11:24:38.752892 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.746285    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:38.792830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.792804    2557 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 11:24:38.794086 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.794057    2557 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 11:24:38.794166 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.794094    2557 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 11:24:38.794166 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.794108    2557 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 11:24:38.794166 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.794114    2557 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 11:24:38.794166 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.794148    2557 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 11:24:38.800979 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.800938    2557 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:38.845676 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.845658    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:38.847804 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.847787    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:38.847887 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.847818    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:38.847887 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.847833    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:38.847887 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.847861    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-203.ec2.internal"
May 26 11:24:38.856728 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.856712    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-203.ec2.internal"
May 26 11:24:38.856796 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.856731    2557 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-203.ec2.internal\": node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:38.875910 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.875892    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:38.895189 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.895172    2557 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal"]
May 26 11:24:38.895240 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.895229    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:38.895881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.895868    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:38.895951 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.895894    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:38.895951 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.895905    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:38.896949 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.896935    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:38.897134 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897120    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:38.897200 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897154    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:38.897734 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897707    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:38.897734 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897726    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:38.897734 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897735    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:38.897881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897745    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:38.897881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897747    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:38.897881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.897758    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:38.898712 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.898697    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal"
May 26 11:24:38.898759 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.898731    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:38.899380 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.899364    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:38.899456 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.899396    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:38.899456 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.899411    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:38.923665 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.923646    2557 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-203.ec2.internal\" not found" node="ip-10-0-132-203.ec2.internal"
May 26 11:24:38.927837 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.927822    2557 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-203.ec2.internal\" not found" node="ip-10-0-132-203.ec2.internal"
May 26 11:24:38.976280 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:38.976262    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:38.978497 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.978481    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e94df3695dd98910ec5b47103a7219a0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal\" (UID: \"e94df3695dd98910ec5b47103a7219a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:38.978564 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.978504    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e94df3695dd98910ec5b47103a7219a0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal\" (UID: \"e94df3695dd98910ec5b47103a7219a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:38.978564 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:38.978521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/691b2498c3b1cc38487ba5d11d3c5965-config\") pod \"kube-apiserver-proxy-ip-10-0-132-203.ec2.internal\" (UID: \"691b2498c3b1cc38487ba5d11d3c5965\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.077107 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.077025    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.079263 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.079246    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/691b2498c3b1cc38487ba5d11d3c5965-config\") pod \"kube-apiserver-proxy-ip-10-0-132-203.ec2.internal\" (UID: \"691b2498c3b1cc38487ba5d11d3c5965\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.079303 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.079275    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e94df3695dd98910ec5b47103a7219a0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal\" (UID: \"e94df3695dd98910ec5b47103a7219a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.079303 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.079298    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e94df3695dd98910ec5b47103a7219a0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal\" (UID: \"e94df3695dd98910ec5b47103a7219a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.079368 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.079341    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/691b2498c3b1cc38487ba5d11d3c5965-config\") pod \"kube-apiserver-proxy-ip-10-0-132-203.ec2.internal\" (UID: \"691b2498c3b1cc38487ba5d11d3c5965\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.079412 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.079399    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e94df3695dd98910ec5b47103a7219a0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal\" (UID: \"e94df3695dd98910ec5b47103a7219a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.079450 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.079435    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e94df3695dd98910ec5b47103a7219a0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal\" (UID: \"e94df3695dd98910ec5b47103a7219a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.177655 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.177632    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.225951 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.225926    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.229594 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.229578    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.278013 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.277990    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.378481 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.378441    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.478858 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.478837    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.569132 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.569114    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 11:24:39.569578 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.569241    2557 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:24:39.579821 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.579806    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.599575 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.599550    2557 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:39.676586 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.676538    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 11:24:39.680391 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.680375    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.688668 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.688637    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 11:19:38 +0000 UTC" deadline="2027-11-06 16:50:09.246863743 +0000 UTC"
May 26 11:24:39.688735 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.688668    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12701h25m29.558198721s"
May 26 11:24:39.695816 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.695798    2557 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:39.734382 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:39.734354    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691b2498c3b1cc38487ba5d11d3c5965.slice/crio-0b67c067faf6eb2d057427a6ed791d674fb78c166d201ad9f7a582c42e84fb65 WatchSource:0}: Error finding container 0b67c067faf6eb2d057427a6ed791d674fb78c166d201ad9f7a582c42e84fb65: Status 404 returned error can't find the container with id 0b67c067faf6eb2d057427a6ed791d674fb78c166d201ad9f7a582c42e84fb65
May 26 11:24:39.734612 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:39.734592    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94df3695dd98910ec5b47103a7219a0.slice/crio-f2cb7b72a376c80cee04005b246eb3abe18a6e3ae9c3a1b467f82c1e9baf0f5b WatchSource:0}: Error finding container f2cb7b72a376c80cee04005b246eb3abe18a6e3ae9c3a1b467f82c1e9baf0f5b: Status 404 returned error can't find the container with id f2cb7b72a376c80cee04005b246eb3abe18a6e3ae9c3a1b467f82c1e9baf0f5b
May 26 11:24:39.737818 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.737805    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:24:39.757809 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.757784    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zxxlm"
May 26 11:24:39.769233 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.769213    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zxxlm"
May 26 11:24:39.781007 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.780991    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.795669 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.795652    2557 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:39.797653 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.797619    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal" event={"ID":"e94df3695dd98910ec5b47103a7219a0","Type":"ContainerStarted","Data":"f2cb7b72a376c80cee04005b246eb3abe18a6e3ae9c3a1b467f82c1e9baf0f5b"}
May 26 11:24:39.798532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.798511    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal" event={"ID":"691b2498c3b1cc38487ba5d11d3c5965","Type":"ContainerStarted","Data":"0b67c067faf6eb2d057427a6ed791d674fb78c166d201ad9f7a582c42e84fb65"}
May 26 11:24:39.881993 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:39.881974    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-132-203.ec2.internal\" not found"
May 26 11:24:39.927289 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.927249    2557 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:39.977652 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.977627    2557 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal"
May 26 11:24:39.993118 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.993103    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:39.994389 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:39.994378    2557 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal"
May 26 11:24:40.009031 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.009016    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:40.387984 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.387898    2557 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:40.654233 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.654165    2557 apiserver.go:52] "Watching apiserver"
May 26 11:24:40.663716 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.663696    2557 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 11:24:40.665184 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.665163    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal","openshift-multus/multus-additional-cni-plugins-wkgxh","openshift-network-diagnostics/network-check-target-m5xdx","openshift-network-operator/iptables-alerter-596zc","kube-system/konnectivity-agent-chvcq","kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq","openshift-image-registry/node-ca-t5gk8","openshift-multus/multus-dq4tc","openshift-multus/network-metrics-daemon-t24mf","openshift-ovn-kubernetes/ovnkube-node-zdkwm","openshift-cluster-node-tuning-operator/tuned-j9tf4"]
May 26 11:24:40.667681 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.667654    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:40.667783 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.667724    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:40.672016 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.671837    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.672016 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.671918    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:40.672016 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.671971    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:40.674106 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.674062    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.675705 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.675688    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 11:24:40.675843 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.675823    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x9z9p\""
May 26 11:24:40.676288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.676209    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 11:24:40.676389 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.676331    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.676484 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.676464    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 11:24:40.676557 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.676502    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 11:24:40.677807 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.677784    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 11:24:40.678191 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.678172    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gbdkw\""
May 26 11:24:40.678302 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.678203    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 11:24:40.678302 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.678250    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 11:24:40.678413 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.678356    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 11:24:40.678462 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.678411    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 11:24:40.678602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.678585    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.679026 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.679008    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 11:24:40.679130 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.679067    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 11:24:40.679130 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.679088    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 11:24:40.679243 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.679149    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 11:24:40.679243 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.679195    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 11:24:40.679497 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.679483    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2q7kp\""
May 26 11:24:40.680861 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.680844    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:40.681220 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.681205    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 11:24:40.681293 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.681234    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 11:24:40.683258 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.682311    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:40.683258 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.682330    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sxhc5\""
May 26 11:24:40.683400 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.683360    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2c9rg\""
May 26 11:24:40.683455 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.683444    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 11:24:40.683503 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.683362    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 11:24:40.686130 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.686112    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.686243 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.686225    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.687043 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:40.687154 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-slash\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687154 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687097    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-etc-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687154 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-ovn\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687154 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovnkube-script-lib\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687354 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687191    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-cni-netd\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687354 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687235    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687354 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687280    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.687354 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4bee2d0-150b-4867-b79e-e5a0d7beb916-iptables-alerter-script\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.687354 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-node-log\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687364    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovnkube-config\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-env-overrides\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovn-node-metrics-cert\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687433    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-system-cni-dir\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.687602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687474    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4bee2d0-150b-4867-b79e-e5a0d7beb916-host-slash\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.687602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-device-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.687602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687637    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687661    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-registration-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687687    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-kubelet\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687735    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-systemd-units\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687760    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687808    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdk2\" (UniqueName: \"kubernetes.io/projected/e4bee2d0-150b-4867-b79e-e5a0d7beb916-kube-api-access-cqdk2\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687891    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnftb\" (UniqueName: \"kubernetes.io/projected/deeb388a-9cbe-4327-bfd2-96b184f74d16-kube-api-access-lnftb\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.687931 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.687968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p724d\" (UniqueName: \"kubernetes.io/projected/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-kube-api-access-p724d\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688002    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-run-netns\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-systemd\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-var-lib-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688112    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhmw\" (UniqueName: \"kubernetes.io/projected/08fa3909-7685-4b83-8d8b-53f0016a570b-kube-api-access-8fhmw\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-socket-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688228    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8pc\" (UniqueName: \"kubernetes.io/projected/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-kube-api-access-zl8pc\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688252    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-log-socket\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-cni-bin\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-cnibin\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-os-release\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.688492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-sys-fs\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.689196 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.688585    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.689196 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.689120    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 11:24:40.689196 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.689135    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 11:24:40.689196 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.689146    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 11:24:40.689196 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.689135    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 11:24:40.689440 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.689431    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tbk5m\""
May 26 11:24:40.689705 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.689690    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dl8rh\""
May 26 11:24:40.691011 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.690994    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fdxhl\""
May 26 11:24:40.691209 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.691182    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:40.691684 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.691655    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 11:24:40.770346 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.770312    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:39 +0000 UTC" deadline="2027-12-16 10:02:16.751565244 +0000 UTC"
May 26 11:24:40.770346 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.770340    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13654h37m35.981228249s"
May 26 11:24:40.778557 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.778537    2557 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 11:24:40.789032 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789005    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p724d\" (UniqueName: \"kubernetes.io/projected/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-kube-api-access-p724d\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:40.789156 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789040    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-systemd\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789156 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789067    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-var-lib-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789156 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789108    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.789156 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789135    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-systemd\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789366 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789139    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhmw\" (UniqueName: \"kubernetes.io/projected/08fa3909-7685-4b83-8d8b-53f0016a570b-kube-api-access-8fhmw\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.789366 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789161    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-var-lib-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789366 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789192    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:40.789366 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789209    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-socket-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.789366 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789318    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-socket-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.789366 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789356    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8pc\" (UniqueName: \"kubernetes.io/projected/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-kube-api-access-zl8pc\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789388    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-log-socket\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789418    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-cni-bin\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789449    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-var-lib-kubelet\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789473    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-log-socket\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789483    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-cni-bin\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789494    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-sys-fs\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789535    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-etc-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789592    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-sys-fs\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.789656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789634    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-ovn\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789661    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovnkube-script-lib\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789685    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789691    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysctl-d\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789694    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-ovn\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789716    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-tuned\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789741    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-cni-netd\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789686    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-etc-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789785    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-cni-netd\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789787    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-os-release\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-k8s-cni-cncf-io\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-netns\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd00228a-356f-4547-b466-403f1cedb343-konnectivity-ca\") pod \"konnectivity-agent-chvcq\" (UID: \"cd00228a-356f-4547-b466-403f1cedb343\") " pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789927    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-node-log\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789952    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-env-overrides\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789976    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovn-node-metrics-cert\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.789996    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-node-log\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790045    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-system-cni-dir\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790001    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-system-cni-dir\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790101    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-host\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec1fce1f-f871-4081-8829-c038c14b7fea-host\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790154    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-device-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-modprobe-d\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869bdcfa-d4b3-4092-9daf-0a803deee389-tmp\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5jd\" (UniqueName: \"kubernetes.io/projected/869bdcfa-d4b3-4092-9daf-0a803deee389-kube-api-access-jn5jd\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790256    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-registration-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790268    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovnkube-script-lib\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790282    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790314    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790322    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdk2\" (UniqueName: \"kubernetes.io/projected/e4bee2d0-150b-4867-b79e-e5a0d7beb916-kube-api-access-cqdk2\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790341    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-sys\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790382    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lnftb\" (UniqueName: \"kubernetes.io/projected/deeb388a-9cbe-4327-bfd2-96b184f74d16-kube-api-access-lnftb\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.790884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790415    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790444    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-hostroot\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf9n\" (UniqueName: \"kubernetes.io/projected/f610ed13-1463-4dc3-9769-6f6876ae0716-kube-api-access-rrf9n\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790411    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-env-overrides\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790493    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-kubernetes\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790484    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-device-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790520    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-run-netns\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-conf-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-run\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790600    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-cni-bin\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790661    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-run-netns\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790663    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790705    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08fa3909-7685-4b83-8d8b-53f0016a570b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-kubelet\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790743    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-registration-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790802    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-cnibin\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790849    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-cnibin\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.791580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790877    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-os-release\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-cni-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-lib-modules\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.790979    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08fa3909-7685-4b83-8d8b-53f0016a570b-os-release\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791404    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791453    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-slash\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-multus-certs\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791510    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791537    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4bee2d0-150b-4867-b79e-e5a0d7beb916-iptables-alerter-script\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.791552    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791560    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5knpc\" (UniqueName: \"kubernetes.io/projected/ec1fce1f-f871-4081-8829-c038c14b7fea-kube-api-access-5knpc\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791585    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovnkube-config\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791617    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-slash\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.791632    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:41.29159424 +0000 UTC m=+3.113978463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791819    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4bee2d0-150b-4867-b79e-e5a0d7beb916-host-slash\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791858    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791858    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f610ed13-1463-4dc3-9769-6f6876ae0716-cni-binary-copy\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791911    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-socket-dir-parent\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791927    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4bee2d0-150b-4867-b79e-e5a0d7beb916-host-slash\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-etc-kubernetes\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.791982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec1fce1f-f871-4081-8829-c038c14b7fea-serviceca\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792024    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792058    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovnkube-config\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-system-cni-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-cni-multus\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792134    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-run-openvswitch\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-daemon-config\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792190    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4bee2d0-150b-4867-b79e-e5a0d7beb916-iptables-alerter-script\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792193    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysconfig\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792240    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-systemd\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792264    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd00228a-356f-4547-b466-403f1cedb343-agent-certs\") pod \"konnectivity-agent-chvcq\" (UID: \"cd00228a-356f-4547-b466-403f1cedb343\") " pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792290    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792316    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.792920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792359    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-kubelet\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792426    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-kubelet\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792447    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792459    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-systemd-units\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792478    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792497    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792532    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792499    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/deeb388a-9cbe-4327-bfd2-96b184f74d16-systemd-units\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-cnibin\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.793526 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.792580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysctl-conf\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.793860 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.793781    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/deeb388a-9cbe-4327-bfd2-96b184f74d16-ovn-node-metrics-cert\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.797559 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.797057    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:40.797559 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.797113    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:40.797559 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.797130    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4hrsb for pod openshift-network-diagnostics/network-check-target-m5xdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:40.797559 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:40.797200    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb podName:5763dd28-8ea6-484b-a856-94899f1552dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:41.297181335 +0000 UTC m=+3.119565560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4hrsb" (UniqueName: "kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb") pod "network-check-target-m5xdx" (UID: "5763dd28-8ea6-484b-a856-94899f1552dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:40.800058 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.799772    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhmw\" (UniqueName: \"kubernetes.io/projected/08fa3909-7685-4b83-8d8b-53f0016a570b-kube-api-access-8fhmw\") pod \"multus-additional-cni-plugins-wkgxh\" (UID: \"08fa3909-7685-4b83-8d8b-53f0016a570b\") " pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:40.800596 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.800574    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnftb\" (UniqueName: \"kubernetes.io/projected/deeb388a-9cbe-4327-bfd2-96b184f74d16-kube-api-access-lnftb\") pod \"ovnkube-node-zdkwm\" (UID: \"deeb388a-9cbe-4327-bfd2-96b184f74d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.801289 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.801262    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdk2\" (UniqueName: \"kubernetes.io/projected/e4bee2d0-150b-4867-b79e-e5a0d7beb916-kube-api-access-cqdk2\") pod \"iptables-alerter-596zc\" (UID: \"e4bee2d0-150b-4867-b79e-e5a0d7beb916\") " pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:40.801418 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.801396    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p724d\" (UniqueName: \"kubernetes.io/projected/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-kube-api-access-p724d\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:40.801542 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.801511    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8pc\" (UniqueName: \"kubernetes.io/projected/9b8653d3-261c-4b99-9bcd-51d7c7ac343b-kube-api-access-zl8pc\") pod \"aws-ebs-csi-driver-node-m4vkq\" (UID: \"9b8653d3-261c-4b99-9bcd-51d7c7ac343b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.893543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893326    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f610ed13-1463-4dc3-9769-6f6876ae0716-cni-binary-copy\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893366    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-socket-dir-parent\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893391    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-etc-kubernetes\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893411    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec1fce1f-f871-4081-8829-c038c14b7fea-serviceca\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.893543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893435    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-system-cni-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893457    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-socket-dir-parent\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893517    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-etc-kubernetes\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893564    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-cni-multus\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893591    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-daemon-config\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893612    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysconfig\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893623    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-system-cni-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893634    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-systemd\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893650    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-cni-multus\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893687    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysconfig\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893705    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-systemd\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893656    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd00228a-356f-4547-b466-403f1cedb343-agent-certs\") pod \"konnectivity-agent-chvcq\" (UID: \"cd00228a-356f-4547-b466-403f1cedb343\") " pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893755    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-cnibin\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893781    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysctl-conf\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893829    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-var-lib-kubelet\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893857    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysctl-d\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893857    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec1fce1f-f871-4081-8829-c038c14b7fea-serviceca\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893921    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-var-lib-kubelet\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893941    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f610ed13-1463-4dc3-9769-6f6876ae0716-cni-binary-copy\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893961    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-cnibin\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.893983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.893983    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysctl-conf\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894032    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-tuned\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894038    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-sysctl-d\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894063    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-os-release\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894114    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-k8s-cni-cncf-io\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894123    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-os-release\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894137    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-netns\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894160    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd00228a-356f-4547-b466-403f1cedb343-konnectivity-ca\") pod \"konnectivity-agent-chvcq\" (UID: \"cd00228a-356f-4547-b466-403f1cedb343\") " pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894168    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-k8s-cni-cncf-io\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894196    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-host\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894220    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec1fce1f-f871-4081-8829-c038c14b7fea-host\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894224    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-netns\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894261    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec1fce1f-f871-4081-8829-c038c14b7fea-host\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894262    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-modprobe-d\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894297    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869bdcfa-d4b3-4092-9daf-0a803deee389-tmp\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894323    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5jd\" (UniqueName: \"kubernetes.io/projected/869bdcfa-d4b3-4092-9daf-0a803deee389-kube-api-access-jn5jd\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894348    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-modprobe-d\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894351    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-sys\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.894881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894342    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-host\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894398    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-hostroot\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894404    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-sys\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894426    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf9n\" (UniqueName: \"kubernetes.io/projected/f610ed13-1463-4dc3-9769-6f6876ae0716-kube-api-access-rrf9n\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894438    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-hostroot\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894456    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-kubernetes\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894482    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-conf-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894508    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-run\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894529    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-cni-bin\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894554    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-kubelet\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894581    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-cni-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894611    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-lib-modules\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894645    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd00228a-356f-4547-b466-403f1cedb343-konnectivity-ca\") pod \"konnectivity-agent-chvcq\" (UID: \"cd00228a-356f-4547-b466-403f1cedb343\") " pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894650    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-multus-certs\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894697    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-kubelet\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894697    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-var-lib-cni-bin\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894697    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5knpc\" (UniqueName: \"kubernetes.io/projected/ec1fce1f-f871-4081-8829-c038c14b7fea-kube-api-access-5knpc\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894737    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-host-run-multus-certs\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.895711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894789    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-cni-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.896498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894798    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-kubernetes\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.896498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894858    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-run\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.896498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894897    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-conf-dir\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.896498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894902    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/869bdcfa-d4b3-4092-9daf-0a803deee389-lib-modules\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.896498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.894936    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f610ed13-1463-4dc3-9769-6f6876ae0716-multus-daemon-config\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.896498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.896272    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd00228a-356f-4547-b466-403f1cedb343-agent-certs\") pod \"konnectivity-agent-chvcq\" (UID: \"cd00228a-356f-4547-b466-403f1cedb343\") " pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:40.896498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.896339    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/869bdcfa-d4b3-4092-9daf-0a803deee389-etc-tuned\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.896886 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.896540    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869bdcfa-d4b3-4092-9daf-0a803deee389-tmp\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.902984 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.902961    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5jd\" (UniqueName: \"kubernetes.io/projected/869bdcfa-d4b3-4092-9daf-0a803deee389-kube-api-access-jn5jd\") pod \"tuned-j9tf4\" (UID: \"869bdcfa-d4b3-4092-9daf-0a803deee389\") " pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:40.903099 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.902973    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5knpc\" (UniqueName: \"kubernetes.io/projected/ec1fce1f-f871-4081-8829-c038c14b7fea-kube-api-access-5knpc\") pod \"node-ca-t5gk8\" (UID: \"ec1fce1f-f871-4081-8829-c038c14b7fea\") " pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:40.903099 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.903034    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf9n\" (UniqueName: \"kubernetes.io/projected/f610ed13-1463-4dc3-9769-6f6876ae0716-kube-api-access-rrf9n\") pod \"multus-dq4tc\" (UID: \"f610ed13-1463-4dc3-9769-6f6876ae0716\") " pod="openshift-multus/multus-dq4tc"
May 26 11:24:40.984217 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.984151    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq"
May 26 11:24:40.992042 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.992019    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:24:40.999634 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:40.999617    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wkgxh"
May 26 11:24:41.005171 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.005152    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-596zc"
May 26 11:24:41.011706 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.011688    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chvcq"
May 26 11:24:41.018225 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.018206    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5gk8"
May 26 11:24:41.024713 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.024696    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dq4tc"
May 26 11:24:41.029221 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.029204    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9tf4"
May 26 11:24:41.272772 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.272713    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1fce1f_f871_4081_8829_c038c14b7fea.slice/crio-3f5ea9a9a44f305a5a939760260d508a9fdaa7d2b2179629360ce10345cddf53 WatchSource:0}: Error finding container 3f5ea9a9a44f305a5a939760260d508a9fdaa7d2b2179629360ce10345cddf53: Status 404 returned error can't find the container with id 3f5ea9a9a44f305a5a939760260d508a9fdaa7d2b2179629360ce10345cddf53
May 26 11:24:41.274802 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.274773    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeb388a_9cbe_4327_bfd2_96b184f74d16.slice/crio-259ac07b55046735d636a1832754d74d37e084ecc92fcecc2b099397adef0000 WatchSource:0}: Error finding container 259ac07b55046735d636a1832754d74d37e084ecc92fcecc2b099397adef0000: Status 404 returned error can't find the container with id 259ac07b55046735d636a1832754d74d37e084ecc92fcecc2b099397adef0000
May 26 11:24:41.276749 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.276676    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869bdcfa_d4b3_4092_9daf_0a803deee389.slice/crio-900131d8af777aba0d80857316e199f54c36c84d1a26c434d8a36dd3ae070ad0 WatchSource:0}: Error finding container 900131d8af777aba0d80857316e199f54c36c84d1a26c434d8a36dd3ae070ad0: Status 404 returned error can't find the container with id 900131d8af777aba0d80857316e199f54c36c84d1a26c434d8a36dd3ae070ad0
May 26 11:24:41.279240 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.279218    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf610ed13_1463_4dc3_9769_6f6876ae0716.slice/crio-af6521d15e1d3c7c558a029f8f212de2e36d51f2bac54e4092142658b3615b44 WatchSource:0}: Error finding container af6521d15e1d3c7c558a029f8f212de2e36d51f2bac54e4092142658b3615b44: Status 404 returned error can't find the container with id af6521d15e1d3c7c558a029f8f212de2e36d51f2bac54e4092142658b3615b44
May 26 11:24:41.280318 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.280290    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fa3909_7685_4b83_8d8b_53f0016a570b.slice/crio-036f16f7d99a78ff725b6769a659501b9f3a757fbb683c579bdde8a280330830 WatchSource:0}: Error finding container 036f16f7d99a78ff725b6769a659501b9f3a757fbb683c579bdde8a280330830: Status 404 returned error can't find the container with id 036f16f7d99a78ff725b6769a659501b9f3a757fbb683c579bdde8a280330830
May 26 11:24:41.280999 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.280900    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8653d3_261c_4b99_9bcd_51d7c7ac343b.slice/crio-bafc268863cc6a47780bf74492e71dfda6b61181fb7bdfd98df4fb55ff268b4a WatchSource:0}: Error finding container bafc268863cc6a47780bf74492e71dfda6b61181fb7bdfd98df4fb55ff268b4a: Status 404 returned error can't find the container with id bafc268863cc6a47780bf74492e71dfda6b61181fb7bdfd98df4fb55ff268b4a
May 26 11:24:41.282288 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.282259    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bee2d0_150b_4867_b79e_e5a0d7beb916.slice/crio-91fb83356d6b5cb77482bf28cd9ee2217a6154242a443726a647a1d92ffbedbc WatchSource:0}: Error finding container 91fb83356d6b5cb77482bf28cd9ee2217a6154242a443726a647a1d92ffbedbc: Status 404 returned error can't find the container with id 91fb83356d6b5cb77482bf28cd9ee2217a6154242a443726a647a1d92ffbedbc
May 26 11:24:41.283822 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:41.283781    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd00228a_356f_4547_b466_403f1cedb343.slice/crio-42574ddd2c7844ca9efc8c472d80b9dd62aa4fcb50f5c4766608ba6d31736837 WatchSource:0}: Error finding container 42574ddd2c7844ca9efc8c472d80b9dd62aa4fcb50f5c4766608ba6d31736837: Status 404 returned error can't find the container with id 42574ddd2c7844ca9efc8c472d80b9dd62aa4fcb50f5c4766608ba6d31736837
May 26 11:24:41.298410 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.298390    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:41.298487 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.298446    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:41.298561 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:41.298542    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:41.298627 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:41.298607    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:42.298588203 +0000 UTC m=+4.120972418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:41.298627 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:41.298609    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:41.298736 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:41.298628    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:41.298736 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:41.298639    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4hrsb for pod openshift-network-diagnostics/network-check-target-m5xdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:41.298736 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:41.298686    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb podName:5763dd28-8ea6-484b-a856-94899f1552dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:42.298669283 +0000 UTC m=+4.121053486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hrsb" (UniqueName: "kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb") pod "network-check-target-m5xdx" (UID: "5763dd28-8ea6-484b-a856-94899f1552dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:41.771487 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.771239    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:39 +0000 UTC" deadline="2028-01-03 15:07:24.384392022 +0000 UTC"
May 26 11:24:41.771487 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.771484    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14091h42m42.612912441s"
May 26 11:24:41.810100 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.795033    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:41.810100 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:41.795194    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:41.810100 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.803759    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chvcq" event={"ID":"cd00228a-356f-4547-b466-403f1cedb343","Type":"ContainerStarted","Data":"42574ddd2c7844ca9efc8c472d80b9dd62aa4fcb50f5c4766608ba6d31736837"}
May 26 11:24:41.810722 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.810674    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-596zc" event={"ID":"e4bee2d0-150b-4867-b79e-e5a0d7beb916","Type":"ContainerStarted","Data":"91fb83356d6b5cb77482bf28cd9ee2217a6154242a443726a647a1d92ffbedbc"}
May 26 11:24:41.820672 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.820630    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq" event={"ID":"9b8653d3-261c-4b99-9bcd-51d7c7ac343b","Type":"ContainerStarted","Data":"bafc268863cc6a47780bf74492e71dfda6b61181fb7bdfd98df4fb55ff268b4a"}
May 26 11:24:41.826975 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.826953    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9tf4" event={"ID":"869bdcfa-d4b3-4092-9daf-0a803deee389","Type":"ContainerStarted","Data":"900131d8af777aba0d80857316e199f54c36c84d1a26c434d8a36dd3ae070ad0"}
May 26 11:24:41.830935 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.830912    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"259ac07b55046735d636a1832754d74d37e084ecc92fcecc2b099397adef0000"}
May 26 11:24:41.836281 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.836258    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal" event={"ID":"691b2498c3b1cc38487ba5d11d3c5965","Type":"ContainerStarted","Data":"ecf8eaa39d57abda873c0e0c29d1717e36e3cbc3f05f7cf8977732050629d36c"}
May 26 11:24:41.841525 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.841505    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerStarted","Data":"036f16f7d99a78ff725b6769a659501b9f3a757fbb683c579bdde8a280330830"}
May 26 11:24:41.858215 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.858192    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dq4tc" event={"ID":"f610ed13-1463-4dc3-9769-6f6876ae0716","Type":"ContainerStarted","Data":"af6521d15e1d3c7c558a029f8f212de2e36d51f2bac54e4092142658b3615b44"}
May 26 11:24:41.859950 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.859929    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5gk8" event={"ID":"ec1fce1f-f871-4081-8829-c038c14b7fea","Type":"ContainerStarted","Data":"3f5ea9a9a44f305a5a939760260d508a9fdaa7d2b2179629360ce10345cddf53"}
May 26 11:24:41.868825 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.868798    2557 generic.go:358] "Generic (PLEG): container finished" podID="e94df3695dd98910ec5b47103a7219a0" containerID="fe73588497ce0b09c698d6f1e585fad168f66e4d0ae3b504b17dbe5c09e911dd" exitCode=0
May 26 11:24:41.868921 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.868838    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal" event={"ID":"e94df3695dd98910ec5b47103a7219a0","Type":"ContainerDied","Data":"fe73588497ce0b09c698d6f1e585fad168f66e4d0ae3b504b17dbe5c09e911dd"}
May 26 11:24:41.885763 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:41.884597    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-203.ec2.internal" podStartSLOduration=1.884583882 podStartE2EDuration="1.884583882s" podCreationTimestamp="2026-05-26 11:24:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:41.851496182 +0000 UTC m=+3.673880412" watchObservedRunningTime="2026-05-26 11:24:41.884583882 +0000 UTC m=+3.706968109"
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:42.307551    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:42.307626    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:42.307753    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:42.307812    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:44.307794071 +0000 UTC m=+6.130178281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:42.308230    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:42.308247    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:42.308259    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4hrsb for pod openshift-network-diagnostics/network-check-target-m5xdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:42.308342 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:42.308309    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb podName:5763dd28-8ea6-484b-a856-94899f1552dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:44.30829175 +0000 UTC m=+6.130676060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hrsb" (UniqueName: "kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb") pod "network-check-target-m5xdx" (UID: "5763dd28-8ea6-484b-a856-94899f1552dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:42.797705 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:42.797207    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:42.797705 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:42.797326    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:42.884397 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:42.884363    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal" event={"ID":"e94df3695dd98910ec5b47103a7219a0","Type":"ContainerStarted","Data":"54068243d48d490e246439ccf84a2a1a4b9134d942a6204a5448db1c0c3f0628"}
May 26 11:24:43.795150 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:43.795120    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:43.795336 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:43.795239    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:44.323908 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:44.323867    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:44.324426 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:44.323938    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:44.324426 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:44.324035    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:44.324426 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:44.324130    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:48.324106204 +0000 UTC m=+10.146490411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:44.324426 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:44.324128    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:44.324426 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:44.324160    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:44.324426 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:44.324173    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4hrsb for pod openshift-network-diagnostics/network-check-target-m5xdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:44.324426 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:44.324211    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb podName:5763dd28-8ea6-484b-a856-94899f1552dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:48.324197899 +0000 UTC m=+10.146582108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hrsb" (UniqueName: "kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb") pod "network-check-target-m5xdx" (UID: "5763dd28-8ea6-484b-a856-94899f1552dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:44.795717 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:44.795173    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:44.795717 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:44.795336    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:45.744408 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.744024    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-203.ec2.internal" podStartSLOduration=6.744004562 podStartE2EDuration="6.744004562s" podCreationTimestamp="2026-05-26 11:24:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:42.898620864 +0000 UTC m=+4.721005091" watchObservedRunningTime="2026-05-26 11:24:45.744004562 +0000 UTC m=+7.566388787"
May 26 11:24:45.744408 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.744405    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-q98hj"]
May 26 11:24:45.748916 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.748467    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.752246 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.751859    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 11:24:45.752246 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.751897    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9q2g2\""
May 26 11:24:45.752246 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.752058    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 11:24:45.794878 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.794857    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:45.794987 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:45.794959    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:45.837811 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.837787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae5a3ccd-890a-464e-a273-c2a2550652f8-hosts-file\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.837927 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.837827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a3ccd-890a-464e-a273-c2a2550652f8-tmp-dir\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.837927 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.837856    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zzf\" (UniqueName: \"kubernetes.io/projected/ae5a3ccd-890a-464e-a273-c2a2550652f8-kube-api-access-x4zzf\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.939224 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.939192    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae5a3ccd-890a-464e-a273-c2a2550652f8-hosts-file\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.939389 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.939236    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a3ccd-890a-464e-a273-c2a2550652f8-tmp-dir\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.939389 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.939266    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zzf\" (UniqueName: \"kubernetes.io/projected/ae5a3ccd-890a-464e-a273-c2a2550652f8-kube-api-access-x4zzf\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.939389 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.939366    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae5a3ccd-890a-464e-a273-c2a2550652f8-hosts-file\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.939686 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.939631    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a3ccd-890a-464e-a273-c2a2550652f8-tmp-dir\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:45.952207 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:45.951169    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zzf\" (UniqueName: \"kubernetes.io/projected/ae5a3ccd-890a-464e-a273-c2a2550652f8-kube-api-access-x4zzf\") pod \"node-resolver-q98hj\" (UID: \"ae5a3ccd-890a-464e-a273-c2a2550652f8\") " pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:46.062189 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:46.062067    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q98hj"
May 26 11:24:46.795029 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:46.794993    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:46.795516 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:46.795148    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:47.794592 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:47.794557    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:47.794754 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:47.794698    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:48.356895 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:48.356849    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:48.357364 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:48.356927    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:48.357364 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:48.357052    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:48.357364 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:48.357132    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:56.357114 +0000 UTC m=+18.179498210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:48.357364 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:48.357225    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:48.357364 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:48.357241    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:48.357364 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:48.357253    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4hrsb for pod openshift-network-diagnostics/network-check-target-m5xdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:48.357364 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:48.357287    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb podName:5763dd28-8ea6-484b-a856-94899f1552dc nodeName:}" failed. No retries permitted until 2026-05-26 11:24:56.357275883 +0000 UTC m=+18.179660087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hrsb" (UniqueName: "kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb") pod "network-check-target-m5xdx" (UID: "5763dd28-8ea6-484b-a856-94899f1552dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:48.795247 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:48.795222    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:48.795422 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:48.795320    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:49.795006 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:49.794980    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:49.795446 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:49.795122    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:50.540126 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.540087    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wndk5"]
May 26 11:24:50.560430 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.560395    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.560560 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:50.560494    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:24:50.673475 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.673443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-kubelet-config\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.673636 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.673507    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-dbus\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.673636 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.673531    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.773894 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.773860    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-kubelet-config\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.774044 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.773940    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-dbus\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.774044 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.773967    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.774044 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.773996    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-kubelet-config\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.774232 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:50.774092    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:50.774232 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:50.774155    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret podName:272e6436-bf5b-4dbc-bd5b-c0b2baf62047 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:51.274134341 +0000 UTC m=+13.096518553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret") pod "global-pull-secret-syncer-wndk5" (UID: "272e6436-bf5b-4dbc-bd5b-c0b2baf62047") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:50.774232 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.774156    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-dbus\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:50.794801 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:50.794743    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:50.794911 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:50.794834    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:51.278037 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:51.277997    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:51.278477 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:51.278159    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:51.278477 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:51.278221    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret podName:272e6436-bf5b-4dbc-bd5b-c0b2baf62047 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:52.278203928 +0000 UTC m=+14.100588133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret") pod "global-pull-secret-syncer-wndk5" (UID: "272e6436-bf5b-4dbc-bd5b-c0b2baf62047") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:51.794634 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:51.794602    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:51.794795 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:51.794742    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:52.287823 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:52.287793    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:52.288267 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:52.287971    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:52.288267 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:52.288047    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret podName:272e6436-bf5b-4dbc-bd5b-c0b2baf62047 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:54.288025095 +0000 UTC m=+16.110409305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret") pod "global-pull-secret-syncer-wndk5" (UID: "272e6436-bf5b-4dbc-bd5b-c0b2baf62047") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:52.798225 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:52.798196    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:52.798378 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:52.798200    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:52.798378 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:52.798300    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:52.798470 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:52.798392    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:24:53.794989 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:53.794954    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:53.795415 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:53.795092    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:54.303211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:54.303178    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:54.303390 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:54.303322    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:54.303461 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:54.303395    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret podName:272e6436-bf5b-4dbc-bd5b-c0b2baf62047 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:58.303377726 +0000 UTC m=+20.125761929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret") pod "global-pull-secret-syncer-wndk5" (UID: "272e6436-bf5b-4dbc-bd5b-c0b2baf62047") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:54.795234 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:54.795202    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:54.795234 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:54.795202    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:54.795721 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:54.795393    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:54.795721 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:54.795489    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:24:55.795180 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:55.795141    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:55.795437 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:55.795277    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:56.419701 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:56.419666    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:56.419870 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:56.419721    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:56.419870 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.419831    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:56.419870 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.419848    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:56.419870 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.419862    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:56.419870 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.419871    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4hrsb for pod openshift-network-diagnostics/network-check-target-m5xdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:56.420057 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.419894    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:12.419877606 +0000 UTC m=+34.242261813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:56.420057 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.419916    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb podName:5763dd28-8ea6-484b-a856-94899f1552dc nodeName:}" failed. No retries permitted until 2026-05-26 11:25:12.419904682 +0000 UTC m=+34.242288889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hrsb" (UniqueName: "kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb") pod "network-check-target-m5xdx" (UID: "5763dd28-8ea6-484b-a856-94899f1552dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:56.798228 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:56.798202    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:56.798643 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:56.798209    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:56.798643 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.798324    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:24:56.798643 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:56.798385    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:57.795180 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:57.795150    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:57.795360 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:57.795255    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:58.335423 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:58.335387    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:58.335831 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:58.335550    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:58.335831 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:58.335614    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret podName:272e6436-bf5b-4dbc-bd5b-c0b2baf62047 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:06.335595293 +0000 UTC m=+28.157979503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret") pod "global-pull-secret-syncer-wndk5" (UID: "272e6436-bf5b-4dbc-bd5b-c0b2baf62047") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:58.791972 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:24:58.791947    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5a3ccd_890a_464e_a273_c2a2550652f8.slice/crio-0fdac7efd60d397902fd6d47497719707f6f9314570b41575190d3c57e13a3d0 WatchSource:0}: Error finding container 0fdac7efd60d397902fd6d47497719707f6f9314570b41575190d3c57e13a3d0: Status 404 returned error can't find the container with id 0fdac7efd60d397902fd6d47497719707f6f9314570b41575190d3c57e13a3d0
May 26 11:24:58.795090 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:58.795046    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:24:58.795180 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:58.795155    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:24:58.795245 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:58.795197    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:24:58.795297 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:58.795260    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:24:58.915398 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:58.915112    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9tf4" event={"ID":"869bdcfa-d4b3-4092-9daf-0a803deee389","Type":"ContainerStarted","Data":"938916be8c2be1431ad10c9e7bcc3bf09e4dbe613106a7751fa70c4dfe8beb24"}
May 26 11:24:58.923969 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:58.923909    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q98hj" event={"ID":"ae5a3ccd-890a-464e-a273-c2a2550652f8","Type":"ContainerStarted","Data":"0fdac7efd60d397902fd6d47497719707f6f9314570b41575190d3c57e13a3d0"}
May 26 11:24:59.795247 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.795066    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:24:59.795926 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:24:59.795343    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:24:59.926343 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.926317    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chvcq" event={"ID":"cd00228a-356f-4547-b466-403f1cedb343","Type":"ContainerStarted","Data":"30dfa6fdf71cd7a455c4ba60da252b87eb5922bdd7caeb294cc1aab8f6188462"}
May 26 11:24:59.927554 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.927528    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-596zc" event={"ID":"e4bee2d0-150b-4867-b79e-e5a0d7beb916","Type":"ContainerStarted","Data":"0f92417af21c286ae54004905528c64cf9efb06f26dee90b8d6a665c257d168e"}
May 26 11:24:59.928798 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.928774    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq" event={"ID":"9b8653d3-261c-4b99-9bcd-51d7c7ac343b","Type":"ContainerStarted","Data":"c2d75046fd7f1c884260bdf259d4dda369351c624bb19f16b098a3e39b77d880"}
May 26 11:24:59.931085 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931053    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:24:59.931430 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931412    2557 generic.go:358] "Generic (PLEG): container finished" podID="deeb388a-9cbe-4327-bfd2-96b184f74d16" containerID="56dfa8e5327af2bb247d45a65e5b24a4c965681dbabef846bf6b89933ca7ae0d" exitCode=1
May 26 11:24:59.931514 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931465    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"8ea0ab2129002c44fa1274777aec569ad5cae2ff97839b8f845390444ecf0098"}
May 26 11:24:59.931514 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931494    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"5259f7ac52a9f2e239dff09f6febee494fff82ca72fd356954fc5d4bd39e47d7"}
May 26 11:24:59.931514 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931507    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"7386611df0e8bb37a764d8bb310086d26852b471a54c938ada87e7ef2a914595"}
May 26 11:24:59.931618 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931522    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"85732dd32cf5cc1d55c6d224b3b7e487209df4df5c0ac8a5568fb73d88a05953"}
May 26 11:24:59.931618 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931535    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerDied","Data":"56dfa8e5327af2bb247d45a65e5b24a4c965681dbabef846bf6b89933ca7ae0d"}
May 26 11:24:59.931618 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.931549    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"f10999a7b48b4012600ce22ccb812b5b35fb88f2a1ddb8099fb0be2851d0b280"}
May 26 11:24:59.932688 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.932663    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q98hj" event={"ID":"ae5a3ccd-890a-464e-a273-c2a2550652f8","Type":"ContainerStarted","Data":"d9f5b2e73fce3c4fb12ed6d5fe3e53aa4fc6f66970413f64908c9c92c673bed9"}
May 26 11:24:59.933872 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.933854    2557 generic.go:358] "Generic (PLEG): container finished" podID="08fa3909-7685-4b83-8d8b-53f0016a570b" containerID="17eb7663ba6dcfe6f6fc5a2e01adeaf65d3cb150777d3ce5a295cdb770d50806" exitCode=0
May 26 11:24:59.933952 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.933908    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerDied","Data":"17eb7663ba6dcfe6f6fc5a2e01adeaf65d3cb150777d3ce5a295cdb770d50806"}
May 26 11:24:59.935349 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.935276    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dq4tc" event={"ID":"f610ed13-1463-4dc3-9769-6f6876ae0716","Type":"ContainerStarted","Data":"87c9f6777511ef99ed4ade340af136ae6bf3d8291f47d308cdc95b8b597876fc"}
May 26 11:24:59.936894 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.936870    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5gk8" event={"ID":"ec1fce1f-f871-4081-8829-c038c14b7fea","Type":"ContainerStarted","Data":"422f7cd4c0b4fa0d326a0d60805f4db28e815fe2cb8b245e0488396a2607a287"}
May 26 11:24:59.952906 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:24:59.952871    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j9tf4" podStartSLOduration=3.446782294 podStartE2EDuration="20.952859345s" podCreationTimestamp="2026-05-26 11:24:39 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.278447235 +0000 UTC m=+3.100831437" lastFinishedPulling="2026-05-26 11:24:58.784524263 +0000 UTC m=+20.606908488" observedRunningTime="2026-05-26 11:24:58.935868434 +0000 UTC m=+20.758252660" watchObservedRunningTime="2026-05-26 11:24:59.952859345 +0000 UTC m=+21.775243569"
May 26 11:25:00.015959 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.015918    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-chvcq" podStartSLOduration=3.51814101 podStartE2EDuration="21.01590528s" podCreationTimestamp="2026-05-26 11:24:39 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.286718438 +0000 UTC m=+3.109102657" lastFinishedPulling="2026-05-26 11:24:58.784482708 +0000 UTC m=+20.606866927" observedRunningTime="2026-05-26 11:24:59.952744703 +0000 UTC m=+21.775128928" watchObservedRunningTime="2026-05-26 11:25:00.01590528 +0000 UTC m=+21.838289559"
May 26 11:25:00.063677 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.063637    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t5gk8" podStartSLOduration=3.55170255 podStartE2EDuration="21.063623709s" podCreationTimestamp="2026-05-26 11:24:39 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.274554568 +0000 UTC m=+3.096938777" lastFinishedPulling="2026-05-26 11:24:58.786475716 +0000 UTC m=+20.608859936" observedRunningTime="2026-05-26 11:25:00.034118931 +0000 UTC m=+21.856503157" watchObservedRunningTime="2026-05-26 11:25:00.063623709 +0000 UTC m=+21.886007934"
May 26 11:25:00.101503 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.101463    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-dq4tc" podStartSLOduration=3.5574853109999998 podStartE2EDuration="21.10144864s" podCreationTimestamp="2026-05-26 11:24:39 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.28140017 +0000 UTC m=+3.103784387" lastFinishedPulling="2026-05-26 11:24:58.825363506 +0000 UTC m=+20.647747716" observedRunningTime="2026-05-26 11:25:00.067349027 +0000 UTC m=+21.889733252" watchObservedRunningTime="2026-05-26 11:25:00.10144864 +0000 UTC m=+21.923832865"
May 26 11:25:00.102140 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.102105    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-q98hj" podStartSLOduration=15.102096103 podStartE2EDuration="15.102096103s" podCreationTimestamp="2026-05-26 11:24:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:25:00.101526448 +0000 UTC m=+21.923910673" watchObservedRunningTime="2026-05-26 11:25:00.102096103 +0000 UTC m=+21.924480325"
May 26 11:25:00.127299 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.127261    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-596zc" podStartSLOduration=3.626813445 podStartE2EDuration="21.127248109s" podCreationTimestamp="2026-05-26 11:24:39 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.286680176 +0000 UTC m=+3.109064383" lastFinishedPulling="2026-05-26 11:24:58.787114835 +0000 UTC m=+20.609499047" observedRunningTime="2026-05-26 11:25:00.127155367 +0000 UTC m=+21.949539591" watchObservedRunningTime="2026-05-26 11:25:00.127248109 +0000 UTC m=+21.949632333"
May 26 11:25:00.179688 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.179665    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 11:25:00.278937 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.278909    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-chvcq"
May 26 11:25:00.279477 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.279458    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-chvcq"
May 26 11:25:00.758535 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.758313    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T11:25:00.179683461Z","UUID":"4dda2003-df31-4e0e-9423-ec306bf16199","Handler":null,"Name":"","Endpoint":""}
May 26 11:25:00.759862 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.759846    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 11:25:00.759969 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.759868    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 11:25:00.794701 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.794672    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:00.794836 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.794709    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:00.794836 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:00.794812    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:25:00.794959 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:00.794927    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:25:00.941639 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:00.941602    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq" event={"ID":"9b8653d3-261c-4b99-9bcd-51d7c7ac343b","Type":"ContainerStarted","Data":"e6998a24f27ae6edc715c395cdebb74565ad721eb4135f05a4b2f4703c90bd57"}
May 26 11:25:01.794578 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:01.794546    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:01.794765 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:01.794662    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:25:01.946643 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:01.946609    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq" event={"ID":"9b8653d3-261c-4b99-9bcd-51d7c7ac343b","Type":"ContainerStarted","Data":"7600cc719f6c7a03874b4ba50e4ddc09b0a5920047f4deb4e67dc5516f68cb01"}
May 26 11:25:01.947182 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:01.946680    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 11:25:02.390188 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:02.390107    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-chvcq"
May 26 11:25:02.390824 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:02.390800    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-chvcq"
May 26 11:25:02.408021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:02.407971    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4vkq" podStartSLOduration=4.542754691 podStartE2EDuration="24.407956845s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.283366485 +0000 UTC m=+3.105750692" lastFinishedPulling="2026-05-26 11:25:01.148568626 +0000 UTC m=+22.970952846" observedRunningTime="2026-05-26 11:25:01.993588046 +0000 UTC m=+23.815972273" watchObservedRunningTime="2026-05-26 11:25:02.407956845 +0000 UTC m=+24.230341071"
May 26 11:25:02.794728 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:02.794702    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:02.794896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:02.794717    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:02.794896 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:02.794815    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:25:02.795015 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:02.794941    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:25:02.950940 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:02.950907    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:25:02.951638 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:02.951518    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"7b7363d42a632a2d895d5add60bef0b6797ef876aedf49befbace3821193d1f0"}
May 26 11:25:03.794753 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:03.794716    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:03.794950 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:03.794880    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:25:04.794866 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.794686    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:04.795432 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.794694    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:04.795432 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:04.794947    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:25:04.795432 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:04.795023    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:25:04.957404 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.957316    2557 generic.go:358] "Generic (PLEG): container finished" podID="08fa3909-7685-4b83-8d8b-53f0016a570b" containerID="3309a561069f2dd908ae2aacf8e8fbe115167023ca5b5fa37278125a05df72ec" exitCode=0
May 26 11:25:04.957404 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.957389    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerDied","Data":"3309a561069f2dd908ae2aacf8e8fbe115167023ca5b5fa37278125a05df72ec"}
May 26 11:25:04.960347 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.960331    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:25:04.960651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.960632    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"d378661e60b6e8c6be5021e44fce246b4b8e93dbce5f545d6814eaaaae9343a0"}
May 26 11:25:04.960913 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.960900    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:25:04.960983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.960926    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:25:04.960983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.960934    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:25:04.961123 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.961110    2557 scope.go:117] "RemoveContainer" containerID="56dfa8e5327af2bb247d45a65e5b24a4c965681dbabef846bf6b89933ca7ae0d"
May 26 11:25:04.976648 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.976626    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:25:04.976817 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:04.976805    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:25:05.795172 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:05.795136    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:05.795515 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:05.795246    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:25:05.964289 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:05.964204    2557 generic.go:358] "Generic (PLEG): container finished" podID="08fa3909-7685-4b83-8d8b-53f0016a570b" containerID="1c1c13585fa1d0905f21fe84d85ff9947d1ab2f478d3552bc6cf9c68c18d12cb" exitCode=0
May 26 11:25:05.964437 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:05.964291    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerDied","Data":"1c1c13585fa1d0905f21fe84d85ff9947d1ab2f478d3552bc6cf9c68c18d12cb"}
May 26 11:25:05.968087 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:05.968056    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:25:05.968439 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:05.968417    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" event={"ID":"deeb388a-9cbe-4327-bfd2-96b184f74d16","Type":"ContainerStarted","Data":"aa9ee263a3ff746417ccaac13c243849e00852f88a3a90331138727d8da748fa"}
May 26 11:25:06.021574 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.021514    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm" podStartSLOduration=10.104836798 podStartE2EDuration="28.021501688s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.27652007 +0000 UTC m=+3.098904278" lastFinishedPulling="2026-05-26 11:24:59.19318496 +0000 UTC m=+21.015569168" observedRunningTime="2026-05-26 11:25:06.021193592 +0000 UTC m=+27.843577817" watchObservedRunningTime="2026-05-26 11:25:06.021501688 +0000 UTC m=+27.843885912"
May 26 11:25:06.302745 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.302681    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m5xdx"]
May 26 11:25:06.302924 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.302908    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:06.303150 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:06.303124    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:25:06.304393 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.304369    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wndk5"]
May 26 11:25:06.304506 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.304462    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:06.304572 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:06.304553    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:25:06.307952 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.307930    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t24mf"]
May 26 11:25:06.308034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.308007    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:06.308128 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:06.308108    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:25:06.397962 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.397925    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:06.398123 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:06.398093    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:25:06.398179 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:06.398156    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret podName:272e6436-bf5b-4dbc-bd5b-c0b2baf62047 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:22.398141962 +0000 UTC m=+44.220526169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret") pod "global-pull-secret-syncer-wndk5" (UID: "272e6436-bf5b-4dbc-bd5b-c0b2baf62047") : object "kube-system"/"original-pull-secret" not registered
May 26 11:25:06.972369 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.972283    2557 generic.go:358] "Generic (PLEG): container finished" podID="08fa3909-7685-4b83-8d8b-53f0016a570b" containerID="4f43e7eb8615eed54cb9470ddda836e98151d6fc5ebbbcbeb90cfd5853d81ae7" exitCode=0
May 26 11:25:06.972713 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:06.972367    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerDied","Data":"4f43e7eb8615eed54cb9470ddda836e98151d6fc5ebbbcbeb90cfd5853d81ae7"}
May 26 11:25:07.794463 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:07.794435    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:07.794615 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:07.794434    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:07.794615 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:07.794539    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:25:07.794724 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:07.794639    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:25:08.797922 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:08.797891    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:08.798495 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:08.798002    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:25:09.795230 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:09.795197    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:09.795386 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:09.795204    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:09.795386 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:09.795299    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:25:09.795507 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:09.795382    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:25:10.797880 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:10.797850    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:10.798521 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:10.797976    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:25:11.795128 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:11.795097    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:11.795379 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:11.795139    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:11.795379 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:11.795222    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m5xdx" podUID="5763dd28-8ea6-484b-a856-94899f1552dc"
May 26 11:25:11.795379 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:11.795346    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wndk5" podUID="272e6436-bf5b-4dbc-bd5b-c0b2baf62047"
May 26 11:25:12.442624 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.442590    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:12.443052 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.442669    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:12.443052 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.442738    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:12.443052 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.442756    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:12.443052 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.442765    2557 projected.go:194] Error preparing data for projected volume kube-api-access-4hrsb for pod openshift-network-diagnostics/network-check-target-m5xdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:12.443052 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.442772    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:12.443052 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.442811    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb podName:5763dd28-8ea6-484b-a856-94899f1552dc nodeName:}" failed. No retries permitted until 2026-05-26 11:25:44.442797287 +0000 UTC m=+66.265181494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hrsb" (UniqueName: "kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb") pod "network-check-target-m5xdx" (UID: "5763dd28-8ea6-484b-a856-94899f1552dc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:12.443052 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.442826    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:44.442819301 +0000 UTC m=+66.265203504 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:12.528779 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.528754    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-132-203.ec2.internal" event="NodeReady"
May 26 11:25:12.528900 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.528892    2557 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 11:25:12.572493 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.572471    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f654b6cd-s6jmw"]
May 26 11:25:12.592582 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.592556    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"]
May 26 11:25:12.592750 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.592730    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.595503 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.595471    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 11:25:12.596115 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.596064    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rrl2s\""
May 26 11:25:12.596313 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.596066    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 11:25:12.596435 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.596410    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 11:25:12.605360 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.605301    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 11:25:12.608654 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.608635    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"]
May 26 11:25:12.608780 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.608765    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:12.611609 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.611594    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mlcjf\""
May 26 11:25:12.611824 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.611806    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 26 11:25:12.611919 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.611858    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 26 11:25:12.625143 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.625123    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"]
May 26 11:25:12.625268 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.625252    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.628529 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.628507    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 26 11:25:12.628759 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.628733    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 26 11:25:12.628759 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.628737    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 26 11:25:12.628876 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.628781    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 26 11:25:12.639626 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.639609    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"]
May 26 11:25:12.639734 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.639721    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.642425 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.642408    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 26 11:25:12.642612 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.642588    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6ctwn\""
May 26 11:25:12.654061 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.654020    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9fbqd"]
May 26 11:25:12.654178 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.654162    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.657092 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.657056    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 26 11:25:12.657475 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.657455    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 26 11:25:12.657585 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.657489    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 26 11:25:12.657585 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.657497    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 26 11:25:12.668662 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.668645    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f654b6cd-s6jmw"]
May 26 11:25:12.668662 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.668665    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"]
May 26 11:25:12.668767 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.668675    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"]
May 26 11:25:12.668767 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.668683    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"]
May 26 11:25:12.668767 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.668690    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"]
May 26 11:25:12.668767 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.668699    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9v24s"]
May 26 11:25:12.668905 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.668768    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.671983 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.671966    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 11:25:12.672240 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.672222    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 11:25:12.672297 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.672251    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9pgx\""
May 26 11:25:12.682920 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.682905    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fbqd"]
May 26 11:25:12.682975 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.682922    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9v24s"]
May 26 11:25:12.683009 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.682999    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:12.686165 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.686144    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 11:25:12.686243 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.686173    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 11:25:12.686243 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.686176    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zx2q6\""
May 26 11:25:12.686311 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.686186    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 11:25:12.744495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.744599 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744505    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-certificates\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.744599 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744523    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-trusted-ca\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.744599 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744539    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-bound-sa-token\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.744599 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744591    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3f98f24f-77ce-4dd4-b266-efb841e0a16f-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.744721 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744631    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c70ccdc-1014-47a3-8233-a61f39785ea4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d88cc5886-hmbd8\" (UID: \"2c70ccdc-1014-47a3-8233-a61f39785ea4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.744721 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-installation-pull-secrets\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.744721 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqtx\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-kube-api-access-8qqtx\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.744721 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s72x\" (UniqueName: \"kubernetes.io/projected/2c70ccdc-1014-47a3-8233-a61f39785ea4-kube-api-access-7s72x\") pod \"managed-serviceaccount-addon-agent-5d88cc5886-hmbd8\" (UID: \"2c70ccdc-1014-47a3-8233-a61f39785ea4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.744721 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744711    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-image-registry-private-configuration\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744732    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-ca\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744747    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-hub\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvr4\" (UniqueName: \"kubernetes.io/projected/3f98f24f-77ce-4dd4-b266-efb841e0a16f-kube-api-access-llvr4\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744825    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61089709-9697-43cb-bf80-a0d676d4511c-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744846    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6d55b72-2856-4c8a-b33d-04403fffd805-ca-trust-extracted\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744873    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744888    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5l6\" (UniqueName: \"kubernetes.io/projected/c413d48d-2421-4f9b-bcfb-5808b3c54007-kube-api-access-ql5l6\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.744904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.745197 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.744936    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.745197 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.745014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c413d48d-2421-4f9b-bcfb-5808b3c54007-tmp\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.745197 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.745062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c413d48d-2421-4f9b-bcfb-5808b3c54007-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.795379 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.795360    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:12.799251 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.799100    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:12.799251 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.799104    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-75s55\""
May 26 11:25:12.846108 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.846211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846125    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.846211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846155    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c413d48d-2421-4f9b-bcfb-5808b3c54007-tmp\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.846211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846180    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.846211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846198    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-trusted-ca\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.846335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846222    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-bound-sa-token\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.846335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846238    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3f98f24f-77ce-4dd4-b266-efb841e0a16f-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.846335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:12.846335 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.846262    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:12.846335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846274    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-installation-pull-secrets\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.846335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846303    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c70ccdc-1014-47a3-8233-a61f39785ea4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d88cc5886-hmbd8\" (UID: \"2c70ccdc-1014-47a3-8233-a61f39785ea4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.846335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846327    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-hub\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846353    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7s72x\" (UniqueName: \"kubernetes.io/projected/2c70ccdc-1014-47a3-8233-a61f39785ea4-kube-api-access-7s72x\") pod \"managed-serviceaccount-addon-agent-5d88cc5886-hmbd8\" (UID: \"2c70ccdc-1014-47a3-8233-a61f39785ea4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846389    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-llvr4\" (UniqueName: \"kubernetes.io/projected/3f98f24f-77ce-4dd4-b266-efb841e0a16f-kube-api-access-llvr4\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846421    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61089709-9697-43cb-bf80-a0d676d4511c-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgdd\" (UniqueName: \"kubernetes.io/projected/f9e2f963-efe8-4935-987d-0661094ac33f-kube-api-access-hkgdd\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846475    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6d55b72-2856-4c8a-b33d-04403fffd805-ca-trust-extracted\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846500    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vh4\" (UniqueName: \"kubernetes.io/projected/89d04d62-aed1-4b37-93fe-170fafdb6e2d-kube-api-access-n9vh4\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846514    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c413d48d-2421-4f9b-bcfb-5808b3c54007-tmp\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846527    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.846278    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846568    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5l6\" (UniqueName: \"kubernetes.io/projected/c413d48d-2421-4f9b-bcfb-5808b3c54007-kube-api-access-ql5l6\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.846598    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:13.346578302 +0000 UTC m=+35.168962517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:25:12.846651 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846632    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846663    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c413d48d-2421-4f9b-bcfb-5808b3c54007-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846691    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d04d62-aed1-4b37-93fe-170fafdb6e2d-tmp-dir\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846720    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-certificates\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846754    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqtx\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-kube-api-access-8qqtx\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846779    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-ca\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846804    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89d04d62-aed1-4b37-93fe-170fafdb6e2d-config-volume\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.846830    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-image-registry-private-configuration\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.847099    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3f98f24f-77ce-4dd4-b266-efb841e0a16f-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.847146    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6d55b72-2856-4c8a-b33d-04403fffd805-ca-trust-extracted\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.847290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.847226    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-trusted-ca\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.847767 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.847470    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61089709-9697-43cb-bf80-a0d676d4511c-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:12.847767 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.847580    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-certificates\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.850310 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.846599    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:25:12.850410 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.850350    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:25:13.350332265 +0000 UTC m=+35.172716468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:25:12.850410 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.850222    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-image-registry-private-configuration\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.850410 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.850402    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-ca\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.850546 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.850421    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c413d48d-2421-4f9b-bcfb-5808b3c54007-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.850546 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.850435    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.850546 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.850402    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-installation-pull-secrets\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.850662 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.850644    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.850952 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.850933    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c70ccdc-1014-47a3-8233-a61f39785ea4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d88cc5886-hmbd8\" (UID: \"2c70ccdc-1014-47a3-8233-a61f39785ea4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.852487 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.852467    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3f98f24f-77ce-4dd4-b266-efb841e0a16f-hub\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.859015 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.858992    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqtx\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-kube-api-access-8qqtx\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.859468 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.859442    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5l6\" (UniqueName: \"kubernetes.io/projected/c413d48d-2421-4f9b-bcfb-5808b3c54007-kube-api-access-ql5l6\") pod \"klusterlet-addon-workmgr-5f88894c94-bg2rm\" (UID: \"c413d48d-2421-4f9b-bcfb-5808b3c54007\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.859468 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.859455    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-bound-sa-token\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:12.860126 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.860105    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s72x\" (UniqueName: \"kubernetes.io/projected/2c70ccdc-1014-47a3-8233-a61f39785ea4-kube-api-access-7s72x\") pod \"managed-serviceaccount-addon-agent-5d88cc5886-hmbd8\" (UID: \"2c70ccdc-1014-47a3-8233-a61f39785ea4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.860532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.860507    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvr4\" (UniqueName: \"kubernetes.io/projected/3f98f24f-77ce-4dd4-b266-efb841e0a16f-kube-api-access-llvr4\") pod \"cluster-proxy-proxy-agent-cc47479fd-ftbsq\" (UID: \"3f98f24f-77ce-4dd4-b266-efb841e0a16f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.933405 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.933377    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:12.947808 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.947779    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:12.947918 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.947876    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgdd\" (UniqueName: \"kubernetes.io/projected/f9e2f963-efe8-4935-987d-0661094ac33f-kube-api-access-hkgdd\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:12.947978 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.947912    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vh4\" (UniqueName: \"kubernetes.io/projected/89d04d62-aed1-4b37-93fe-170fafdb6e2d-kube-api-access-n9vh4\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.947978 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.947946    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:12.947978 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.947965    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d04d62-aed1-4b37-93fe-170fafdb6e2d-tmp-dir\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.948115 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.948013    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:13.447992569 +0000 UTC m=+35.270376783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:25:12.948115 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.948038    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89d04d62-aed1-4b37-93fe-170fafdb6e2d-config-volume\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.948115 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.948098    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.948263 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.948202    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:12.948263 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:12.948240    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:25:13.44822637 +0000 UTC m=+35.270610589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:25:12.948368 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.948328    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d04d62-aed1-4b37-93fe-170fafdb6e2d-tmp-dir\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.948657 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.948637    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89d04d62-aed1-4b37-93fe-170fafdb6e2d-config-volume\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.959034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.959019    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"
May 26 11:25:12.962897 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.962878    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgdd\" (UniqueName: \"kubernetes.io/projected/f9e2f963-efe8-4935-987d-0661094ac33f-kube-api-access-hkgdd\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:12.963540 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.963522    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vh4\" (UniqueName: \"kubernetes.io/projected/89d04d62-aed1-4b37-93fe-170fafdb6e2d-kube-api-access-n9vh4\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:12.965514 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.965397    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:25:12.988910 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:12.988858    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerStarted","Data":"9caf614fde356d676847efa3aeaedb90d11f9d8cb28e4026b915ac9c96fcda15"}
May 26 11:25:13.133515 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.133486    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"]
May 26 11:25:13.137196 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.137172    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8"]
May 26 11:25:13.142114 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.142058    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"]
May 26 11:25:13.145845 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:25:13.145818    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c70ccdc_1014_47a3_8233_a61f39785ea4.slice/crio-46dd41ef28c4bb0a1cb9f967d5d65a50b488b06c2c573437954cf673e4f2f0ef WatchSource:0}: Error finding container 46dd41ef28c4bb0a1cb9f967d5d65a50b488b06c2c573437954cf673e4f2f0ef: Status 404 returned error can't find the container with id 46dd41ef28c4bb0a1cb9f967d5d65a50b488b06c2c573437954cf673e4f2f0ef
May 26 11:25:13.146371 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:25:13.146333    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f98f24f_77ce_4dd4_b266_efb841e0a16f.slice/crio-20c8c270c47da99987ef85f45242729d6b9bee8de2f70f328c45e21db432e76f WatchSource:0}: Error finding container 20c8c270c47da99987ef85f45242729d6b9bee8de2f70f328c45e21db432e76f: Status 404 returned error can't find the container with id 20c8c270c47da99987ef85f45242729d6b9bee8de2f70f328c45e21db432e76f
May 26 11:25:13.147155 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:25:13.147128    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc413d48d_2421_4f9b_bcfb_5808b3c54007.slice/crio-3f1c54d153eca60407eb8c02552c82069dc9dab34f389952169d30048135e4b2 WatchSource:0}: Error finding container 3f1c54d153eca60407eb8c02552c82069dc9dab34f389952169d30048135e4b2: Status 404 returned error can't find the container with id 3f1c54d153eca60407eb8c02552c82069dc9dab34f389952169d30048135e4b2
May 26 11:25:13.353171 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.353139    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:13.353327 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.353195    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:13.353327 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.353281    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:25:13.353413 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.353344    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.353329245 +0000 UTC m=+36.175713448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:25:13.353413 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.353290    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:13.353413 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.353376    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:25:13.353504 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.353427    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.353413641 +0000 UTC m=+36.175797861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:25:13.453750 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.453670    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:13.454136 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.453834    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:13.454136 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.453839    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:13.454136 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.453895    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:13.454136 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.453902    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.453883321 +0000 UTC m=+36.276267546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:25:13.454136 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:13.453951    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.45393895 +0000 UTC m=+36.276323152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:25:13.794674 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.794645    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:13.794900 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.794685    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:13.798658 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.798633    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 11:25:13.798877 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.798859    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:13.799232 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.799158    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:13.799791 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.799773    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l6xzk\""
May 26 11:25:13.992275 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.992225    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" event={"ID":"3f98f24f-77ce-4dd4-b266-efb841e0a16f","Type":"ContainerStarted","Data":"20c8c270c47da99987ef85f45242729d6b9bee8de2f70f328c45e21db432e76f"}
May 26 11:25:13.995316 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.995254    2557 generic.go:358] "Generic (PLEG): container finished" podID="08fa3909-7685-4b83-8d8b-53f0016a570b" containerID="9caf614fde356d676847efa3aeaedb90d11f9d8cb28e4026b915ac9c96fcda15" exitCode=0
May 26 11:25:13.995448 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.995330    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerDied","Data":"9caf614fde356d676847efa3aeaedb90d11f9d8cb28e4026b915ac9c96fcda15"}
May 26 11:25:13.997516 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.997484    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8" event={"ID":"2c70ccdc-1014-47a3-8233-a61f39785ea4","Type":"ContainerStarted","Data":"46dd41ef28c4bb0a1cb9f967d5d65a50b488b06c2c573437954cf673e4f2f0ef"}
May 26 11:25:13.998943 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:13.998908    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm" event={"ID":"c413d48d-2421-4f9b-bcfb-5808b3c54007","Type":"ContainerStarted","Data":"3f1c54d153eca60407eb8c02552c82069dc9dab34f389952169d30048135e4b2"}
May 26 11:25:14.361774 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:14.360840    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:14.361774 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:14.360992    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:14.361774 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.361140    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:25:14.361774 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.361218    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:25:16.361199009 +0000 UTC m=+38.183583213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:25:14.361774 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.361636    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:14.361774 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.361652    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:25:14.361774 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.361696    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:16.361679993 +0000 UTC m=+38.184064200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:25:14.462335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:14.461551    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:14.462335 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:14.461667    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:14.462335 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.461824    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:14.462335 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.461881    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:25:16.461863919 +0000 UTC m=+38.284248127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:25:14.462335 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.462256    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:14.462335 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:14.462304    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:16.462288601 +0000 UTC m=+38.284672807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:25:15.007137 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:15.006928    2557 generic.go:358] "Generic (PLEG): container finished" podID="08fa3909-7685-4b83-8d8b-53f0016a570b" containerID="c45d04f38ae7b5f857d7d4f58d01c1743c1085f30a8640b894c9f201e9c297a5" exitCode=0
May 26 11:25:15.007137 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:15.006993    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerDied","Data":"c45d04f38ae7b5f857d7d4f58d01c1743c1085f30a8640b894c9f201e9c297a5"}
May 26 11:25:16.379761 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:16.379580    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:16.379761 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.379754    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:16.380248 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.379779    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:25:16.380248 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:16.379827    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:16.380248 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.379846    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:20.379825677 +0000 UTC m=+42.202209895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:25:16.380248 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.379892    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:25:16.380248 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.379944    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:25:20.379928074 +0000 UTC m=+42.202312279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:25:16.480788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:16.480756    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:16.480969 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:16.480857    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:16.480969 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.480900    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:16.480969 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.480961    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:16.481128 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.480973    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:25:20.480953183 +0000 UTC m=+42.303337387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:25:16.481128 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:16.481008    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:20.480993224 +0000 UTC m=+42.303377442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:25:20.414102 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:20.414058    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:20.414377 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.414197    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:20.414377 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.414214    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:25:20.414377 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.414262    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:28.414247316 +0000 UTC m=+50.236631519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:25:20.414377 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:20.414258    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:20.414377 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.414377    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:25:20.414543 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.414423    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:25:28.414409047 +0000 UTC m=+50.236793272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:25:20.515559 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:20.515525    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:20.515722 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:20.515592    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:20.515722 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.515640    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:20.515722 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.515663    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:20.515722 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.515711    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:25:28.515693164 +0000 UTC m=+50.338077389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:25:20.515873 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:20.515729    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:28.515720603 +0000 UTC m=+50.338104806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:25:21.022998 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.022924    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" event={"ID":"08fa3909-7685-4b83-8d8b-53f0016a570b","Type":"ContainerStarted","Data":"fecd1167e4c98a1d686e14ee8f48c653fd1e784acc2f250ed8b6d9a46291bfb8"}
May 26 11:25:21.024225 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.024199    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8" event={"ID":"2c70ccdc-1014-47a3-8233-a61f39785ea4","Type":"ContainerStarted","Data":"005f0b37f37a67aa591447e06d782b7d464f06c33d9dfea6c3b5810c9380e51b"}
May 26 11:25:21.025390 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.025368    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm" event={"ID":"c413d48d-2421-4f9b-bcfb-5808b3c54007","Type":"ContainerStarted","Data":"278717fb7a57c8f94c42a5b482a44d06c222b6ca762aa2b470d0cc5c37c1bfbe"}
May 26 11:25:21.025549 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.025536    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:21.026702 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.026682    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" event={"ID":"3f98f24f-77ce-4dd4-b266-efb841e0a16f","Type":"ContainerStarted","Data":"e4e4fd62794a88abe2b13ea95b025b6676a973187a034420b9b1c8a66c822c21"}
May 26 11:25:21.027158 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.027140    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:25:21.046834 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.046792    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wkgxh" podStartSLOduration=10.561630013 podStartE2EDuration="42.046779728s" podCreationTimestamp="2026-05-26 11:24:39 +0000 UTC" firstStartedPulling="2026-05-26 11:24:41.282781737 +0000 UTC m=+3.105165940" lastFinishedPulling="2026-05-26 11:25:12.767931433 +0000 UTC m=+34.590315655" observedRunningTime="2026-05-26 11:25:21.045169738 +0000 UTC m=+42.867553976" watchObservedRunningTime="2026-05-26 11:25:21.046779728 +0000 UTC m=+42.869163953"
May 26 11:25:21.063155 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.063116    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8" podStartSLOduration=14.515749199 podStartE2EDuration="22.063106505s" podCreationTimestamp="2026-05-26 11:24:59 +0000 UTC" firstStartedPulling="2026-05-26 11:25:13.148327609 +0000 UTC m=+34.970711819" lastFinishedPulling="2026-05-26 11:25:20.695684902 +0000 UTC m=+42.518069125" observedRunningTime="2026-05-26 11:25:21.06297813 +0000 UTC m=+42.885362356" watchObservedRunningTime="2026-05-26 11:25:21.063106505 +0000 UTC m=+42.885490729"
May 26 11:25:21.082577 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:21.082542    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm" podStartSLOduration=14.518640229 podStartE2EDuration="22.082531841s" podCreationTimestamp="2026-05-26 11:24:59 +0000 UTC" firstStartedPulling="2026-05-26 11:25:13.148998888 +0000 UTC m=+34.971383092" lastFinishedPulling="2026-05-26 11:25:20.712890501 +0000 UTC m=+42.535274704" observedRunningTime="2026-05-26 11:25:21.082223551 +0000 UTC m=+42.904607789" watchObservedRunningTime="2026-05-26 11:25:21.082531841 +0000 UTC m=+42.904916083"
May 26 11:25:22.430766 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:22.430733    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:22.434004 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:22.433978    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/272e6436-bf5b-4dbc-bd5b-c0b2baf62047-original-pull-secret\") pod \"global-pull-secret-syncer-wndk5\" (UID: \"272e6436-bf5b-4dbc-bd5b-c0b2baf62047\") " pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:22.528785 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:22.528759    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wndk5"
May 26 11:25:22.653546 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:22.653483    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wndk5"]
May 26 11:25:22.657686 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:25:22.657658    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272e6436_bf5b_4dbc_bd5b_c0b2baf62047.slice/crio-9ca23926edbc233957df0961984f7626abce4c248066370535052f30ec526510 WatchSource:0}: Error finding container 9ca23926edbc233957df0961984f7626abce4c248066370535052f30ec526510: Status 404 returned error can't find the container with id 9ca23926edbc233957df0961984f7626abce4c248066370535052f30ec526510
May 26 11:25:23.030589 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:23.030550    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wndk5" event={"ID":"272e6436-bf5b-4dbc-bd5b-c0b2baf62047","Type":"ContainerStarted","Data":"9ca23926edbc233957df0961984f7626abce4c248066370535052f30ec526510"}
May 26 11:25:25.036419 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:25.036386    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" event={"ID":"3f98f24f-77ce-4dd4-b266-efb841e0a16f","Type":"ContainerStarted","Data":"6af9745ab05b72063a1493121e7b8775cd1553a01682f13b307d5b2dc8902a71"}
May 26 11:25:25.036419 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:25.036423    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" event={"ID":"3f98f24f-77ce-4dd4-b266-efb841e0a16f","Type":"ContainerStarted","Data":"511e81411e1a6e9d561b1e67e8d08fdd84b677c8c46ba72910429f2e2cad4058"}
May 26 11:25:25.057597 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:25.057556    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" podStartSLOduration=15.070672728 podStartE2EDuration="26.057543058s" podCreationTimestamp="2026-05-26 11:24:59 +0000 UTC" firstStartedPulling="2026-05-26 11:25:13.14838522 +0000 UTC m=+34.970769428" lastFinishedPulling="2026-05-26 11:25:24.135255546 +0000 UTC m=+45.957639758" observedRunningTime="2026-05-26 11:25:25.055603167 +0000 UTC m=+46.877987392" watchObservedRunningTime="2026-05-26 11:25:25.057543058 +0000 UTC m=+46.879927283"
May 26 11:25:27.042412 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:27.042378    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wndk5" event={"ID":"272e6436-bf5b-4dbc-bd5b-c0b2baf62047","Type":"ContainerStarted","Data":"c03cfbe8d571d3183c601efd36081175c8f3789ef4c3de78b1f0636e929e5f38"}
May 26 11:25:27.061184 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:27.061142    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wndk5" podStartSLOduration=33.107402267 podStartE2EDuration="37.061129147s" podCreationTimestamp="2026-05-26 11:24:50 +0000 UTC" firstStartedPulling="2026-05-26 11:25:22.660049293 +0000 UTC m=+44.482433501" lastFinishedPulling="2026-05-26 11:25:26.613776177 +0000 UTC m=+48.436160381" observedRunningTime="2026-05-26 11:25:27.060223319 +0000 UTC m=+48.882607543" watchObservedRunningTime="2026-05-26 11:25:27.061129147 +0000 UTC m=+48.883513395"
May 26 11:25:28.477445 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:28.477407    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:28.477812 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:28.477487    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:28.477812 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.477547    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:28.477812 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.477567    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:25:28.477812 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.477580    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:25:28.477812 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.477622    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:44.477607556 +0000 UTC m=+66.299991758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:25:28.477812 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.477635    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:25:44.477629152 +0000 UTC m=+66.300013355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:25:28.578112 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:28.578067    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:28.578244 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:28.578159    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:28.578244 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.578205    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:28.578244 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.578241    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:28.578341 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.578263    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:44.578249461 +0000 UTC m=+66.400633664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:25:28.578341 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:28.578278    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:25:44.578271297 +0000 UTC m=+66.400655499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:25:36.990465 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:36.990436    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdkwm"
May 26 11:25:44.493857 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.493822    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:25:44.493857 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.493862    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:25:44.494275 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.493893    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:44.494275 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.493925    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:25:44.494275 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.493958    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:44.494275 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.493975    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:25:44.494275 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.494008    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:25:44.494275 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.494027    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:16.494011975 +0000 UTC m=+98.316396179 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:25:44.494275 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.494057    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:26:16.494042038 +0000 UTC m=+98.316426250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:25:44.496726 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.496710    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:44.496885 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.496873    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:44.504814 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.504796    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:25:44.504865 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.504838    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:48.504826196 +0000 UTC m=+130.327210399 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : secret "metrics-daemon-secret" not found
May 26 11:25:44.508266 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.508250    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:44.517143 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.517116    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrsb\" (UniqueName: \"kubernetes.io/projected/5763dd28-8ea6-484b-a856-94899f1552dc-kube-api-access-4hrsb\") pod \"network-check-target-m5xdx\" (UID: \"5763dd28-8ea6-484b-a856-94899f1552dc\") " pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:44.594253 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.594224    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:25:44.594341 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.594283    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:25:44.594389 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.594374    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:44.594427 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.594407    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:44.594459 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.594428    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:26:16.594412908 +0000 UTC m=+98.416797119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:25:44.594459 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:25:44.594446    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:26:16.59443646 +0000 UTC m=+98.416820670 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:25:44.728707 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.728685    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l6xzk\""
May 26 11:25:44.733388 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.733374    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:44.853614 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:44.853578    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m5xdx"]
May 26 11:25:44.857206 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:25:44.857176    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5763dd28_8ea6_484b_a856_94899f1552dc.slice/crio-515be08075ad7c030330af055634ba0f278a4a22cdbe649b5a1a9479b6f0812b WatchSource:0}: Error finding container 515be08075ad7c030330af055634ba0f278a4a22cdbe649b5a1a9479b6f0812b: Status 404 returned error can't find the container with id 515be08075ad7c030330af055634ba0f278a4a22cdbe649b5a1a9479b6f0812b
May 26 11:25:45.094791 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:45.094758    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m5xdx" event={"ID":"5763dd28-8ea6-484b-a856-94899f1552dc","Type":"ContainerStarted","Data":"515be08075ad7c030330af055634ba0f278a4a22cdbe649b5a1a9479b6f0812b"}
May 26 11:25:48.103780 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:48.103743    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m5xdx" event={"ID":"5763dd28-8ea6-484b-a856-94899f1552dc","Type":"ContainerStarted","Data":"c654832d1476d32dc5b76b52aa2b3ea0af0508fe72a8076e258ab390590f9a91"}
May 26 11:25:48.104231 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:48.103858    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:25:48.121472 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:25:48.121429    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m5xdx" podStartSLOduration=67.267569645 podStartE2EDuration="1m10.121417022s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="2026-05-26 11:25:44.859116856 +0000 UTC m=+66.681501062" lastFinishedPulling="2026-05-26 11:25:47.712964229 +0000 UTC m=+69.535348439" observedRunningTime="2026-05-26 11:25:48.121399083 +0000 UTC m=+69.943783308" watchObservedRunningTime="2026-05-26 11:25:48.121417022 +0000 UTC m=+69.943801241"
May 26 11:26:16.521575 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:26:16.521545    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:26:16.521918 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:26:16.521621    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:26:16.521918 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.521692    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:26:16.521918 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.521709    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:26:16.521918 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.521721    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:26:16.521918 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.521779    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:20.521763482 +0000 UTC m=+162.344147904 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:26:16.521918 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.521794    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:27:20.521787915 +0000 UTC m=+162.344172117 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:26:16.622717 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:26:16.622689    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:26:16.622832 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:26:16.622760    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:26:16.622873 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.622828    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:26:16.622873 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.622868    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:26:16.622933 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.622888    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:27:20.622873159 +0000 UTC m=+162.445257363 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:26:16.622933 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:16.622907    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:27:20.622896325 +0000 UTC m=+162.445280529 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:26:19.108286 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:26:19.108252    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m5xdx"
May 26 11:26:48.555717 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:26:48.555683    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:26:48.556151 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:48.555794    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:26:48.556151 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:26:48.555851    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs podName:d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:50.55583669 +0000 UTC m=+252.378220893 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs") pod "network-metrics-daemon-t24mf" (UID: "d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20") : secret "metrics-daemon-secret" not found
May 26 11:27:15.427339 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:15.427306    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q98hj_ae5a3ccd-890a-464e-a273-c2a2550652f8/dns-node-resolver/0.log"
May 26 11:27:15.604356 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:15.604318    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-f654b6cd-s6jmw" podUID="e6d55b72-2856-4c8a-b33d-04403fffd805"
May 26 11:27:15.617481 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:15.617455    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9" podUID="61089709-9697-43cb-bf80-a0d676d4511c"
May 26 11:27:15.695061 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:15.695003    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9fbqd" podUID="89d04d62-aed1-4b37-93fe-170fafdb6e2d"
May 26 11:27:15.700182 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:15.700154    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-9v24s" podUID="f9e2f963-efe8-4935-987d-0661094ac33f"
May 26 11:27:15.810720 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:15.810696    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-t24mf" podUID="d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20"
May 26 11:27:16.228273 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:16.228251    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5gk8_ec1fce1f-f871-4081-8829-c038c14b7fea/node-ca/0.log"
May 26 11:27:16.304476 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:16.304450    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fbqd"
May 26 11:27:16.304589 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:16.304450    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:27:16.304650 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:16.304450    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:27:20.582627 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:20.582592    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:27:20.583019 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:20.582650    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") pod \"image-registry-f654b6cd-s6jmw\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") " pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:27:20.583019 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.582742    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 11:27:20.583019 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.582810    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert podName:61089709-9697-43cb-bf80-a0d676d4511c nodeName:}" failed. No retries permitted until 2026-05-26 11:29:22.582792586 +0000 UTC m=+284.405176793 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-845q9" (UID: "61089709-9697-43cb-bf80-a0d676d4511c") : secret "networking-console-plugin-cert" not found
May 26 11:27:20.583019 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.582744    2557 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:27:20.583019 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.582842    2557 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f654b6cd-s6jmw: secret "image-registry-tls" not found
May 26 11:27:20.583019 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.582896    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls podName:e6d55b72-2856-4c8a-b33d-04403fffd805 nodeName:}" failed. No retries permitted until 2026-05-26 11:29:22.582880891 +0000 UTC m=+284.405265094 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls") pod "image-registry-f654b6cd-s6jmw" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805") : secret "image-registry-tls" not found
May 26 11:27:20.683328 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:20.683303    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:27:20.683458 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:20.683357    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:27:20.683458 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.683436    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:27:20.683523 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.683480    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls podName:89d04d62-aed1-4b37-93fe-170fafdb6e2d nodeName:}" failed. No retries permitted until 2026-05-26 11:29:22.683467947 +0000 UTC m=+284.505852150 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls") pod "dns-default-9fbqd" (UID: "89d04d62-aed1-4b37-93fe-170fafdb6e2d") : secret "dns-default-metrics-tls" not found
May 26 11:27:20.683562 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.683437    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:27:20.683594 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:20.683575    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert podName:f9e2f963-efe8-4935-987d-0661094ac33f nodeName:}" failed. No retries permitted until 2026-05-26 11:29:22.683563499 +0000 UTC m=+284.505947703 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert") pod "ingress-canary-9v24s" (UID: "f9e2f963-efe8-4935-987d-0661094ac33f") : secret "canary-serving-cert" not found
May 26 11:27:21.026400 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:21.026364    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm" podUID="c413d48d-2421-4f9b-bcfb-5808b3c54007" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused"
May 26 11:27:21.316743 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:21.316713    2557 generic.go:358] "Generic (PLEG): container finished" podID="c413d48d-2421-4f9b-bcfb-5808b3c54007" containerID="278717fb7a57c8f94c42a5b482a44d06c222b6ca762aa2b470d0cc5c37c1bfbe" exitCode=1
May 26 11:27:21.316872 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:21.316785    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm" event={"ID":"c413d48d-2421-4f9b-bcfb-5808b3c54007","Type":"ContainerDied","Data":"278717fb7a57c8f94c42a5b482a44d06c222b6ca762aa2b470d0cc5c37c1bfbe"}
May 26 11:27:21.317130 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:21.317115    2557 scope.go:117] "RemoveContainer" containerID="278717fb7a57c8f94c42a5b482a44d06c222b6ca762aa2b470d0cc5c37c1bfbe"
May 26 11:27:21.321353 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:21.321327    2557 generic.go:358] "Generic (PLEG): container finished" podID="2c70ccdc-1014-47a3-8233-a61f39785ea4" containerID="005f0b37f37a67aa591447e06d782b7d464f06c33d9dfea6c3b5810c9380e51b" exitCode=255
May 26 11:27:21.321431 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:21.321378    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8" event={"ID":"2c70ccdc-1014-47a3-8233-a61f39785ea4","Type":"ContainerDied","Data":"005f0b37f37a67aa591447e06d782b7d464f06c33d9dfea6c3b5810c9380e51b"}
May 26 11:27:21.321671 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:21.321658    2557 scope.go:117] "RemoveContainer" containerID="005f0b37f37a67aa591447e06d782b7d464f06c33d9dfea6c3b5810c9380e51b"
May 26 11:27:22.327549 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:22.327510    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d88cc5886-hmbd8" event={"ID":"2c70ccdc-1014-47a3-8233-a61f39785ea4","Type":"ContainerStarted","Data":"871ff03806083a48704f307dc991d145e0e42a30b8d90bcfbffdb1150c9e2e8b"}
May 26 11:27:22.328889 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:22.328865    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm" event={"ID":"c413d48d-2421-4f9b-bcfb-5808b3c54007","Type":"ContainerStarted","Data":"cc37602bc45bd79d5715ff1ecff96ebf35590b9cb1d80a7a9f2c1b6fff93ab31"}
May 26 11:27:22.329118 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:22.329101    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:27:22.329763 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:22.329749    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f88894c94-bg2rm"
May 26 11:27:25.794429 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:25.794399    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:27:26.794589 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:26.794555    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:27:36.768579 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.768498    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t6gkb"]
May 26 11:27:36.771566 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.771551    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:36.775112 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.775092    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 11:27:36.776287 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.776273    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 11:27:36.776357 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.776293    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 11:27:36.776500 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.776484    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 11:27:36.776702 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.776687    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d2xxs\""
May 26 11:27:36.797759 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.797738    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t6gkb"]
May 26 11:27:36.899997 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.899963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/257582b4-4b4d-4ab5-b7ef-0350b45efad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:36.900172 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.900010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/257582b4-4b4d-4ab5-b7ef-0350b45efad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:36.900172 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.900063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnhn\" (UniqueName: \"kubernetes.io/projected/257582b4-4b4d-4ab5-b7ef-0350b45efad9-kube-api-access-xfnhn\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:36.900172 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.900139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/257582b4-4b4d-4ab5-b7ef-0350b45efad9-data-volume\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:36.900300 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:36.900224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/257582b4-4b4d-4ab5-b7ef-0350b45efad9-crio-socket\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.000831 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.000800    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/257582b4-4b4d-4ab5-b7ef-0350b45efad9-crio-socket\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.000975 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.000842    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/257582b4-4b4d-4ab5-b7ef-0350b45efad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.000975 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.000882    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/257582b4-4b4d-4ab5-b7ef-0350b45efad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.000975 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.000917    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnhn\" (UniqueName: \"kubernetes.io/projected/257582b4-4b4d-4ab5-b7ef-0350b45efad9-kube-api-access-xfnhn\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.000975 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.000914    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/257582b4-4b4d-4ab5-b7ef-0350b45efad9-crio-socket\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.001262 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.001052    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/257582b4-4b4d-4ab5-b7ef-0350b45efad9-data-volume\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.001377 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.001357    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/257582b4-4b4d-4ab5-b7ef-0350b45efad9-data-volume\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.001533 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.001506    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/257582b4-4b4d-4ab5-b7ef-0350b45efad9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.002972 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.002954    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/257582b4-4b4d-4ab5-b7ef-0350b45efad9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.021538 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.021477    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnhn\" (UniqueName: \"kubernetes.io/projected/257582b4-4b4d-4ab5-b7ef-0350b45efad9-kube-api-access-xfnhn\") pod \"insights-runtime-extractor-t6gkb\" (UID: \"257582b4-4b4d-4ab5-b7ef-0350b45efad9\") " pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.080609 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.080569    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t6gkb"
May 26 11:27:37.197899 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.197871    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t6gkb"]
May 26 11:27:37.200808 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:27:37.200785    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257582b4_4b4d_4ab5_b7ef_0350b45efad9.slice/crio-e358e1c2858b4d21165dd81233e7f39828f068e9c4b8c656d1fb231c9ff895fc WatchSource:0}: Error finding container e358e1c2858b4d21165dd81233e7f39828f068e9c4b8c656d1fb231c9ff895fc: Status 404 returned error can't find the container with id e358e1c2858b4d21165dd81233e7f39828f068e9c4b8c656d1fb231c9ff895fc
May 26 11:27:37.364320 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.364284    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t6gkb" event={"ID":"257582b4-4b4d-4ab5-b7ef-0350b45efad9","Type":"ContainerStarted","Data":"e795d67f3f55c1a9f63d7aa0cfde8256a0ef6583c680176bd1223c824d528b66"}
May 26 11:27:37.364320 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:37.364321    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t6gkb" event={"ID":"257582b4-4b4d-4ab5-b7ef-0350b45efad9","Type":"ContainerStarted","Data":"e358e1c2858b4d21165dd81233e7f39828f068e9c4b8c656d1fb231c9ff895fc"}
May 26 11:27:38.367750 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:38.367715    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t6gkb" event={"ID":"257582b4-4b4d-4ab5-b7ef-0350b45efad9","Type":"ContainerStarted","Data":"f06eae07cb47c6ebf47c18915f0b48688c1c441373c40e72eac772188bb9dffc"}
May 26 11:27:40.374059 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:40.374018    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t6gkb" event={"ID":"257582b4-4b4d-4ab5-b7ef-0350b45efad9","Type":"ContainerStarted","Data":"8105d7c7208cec9d25078142bfb4250b08e73cae668e22a74f884f3262696d57"}
May 26 11:27:40.414622 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:40.414575    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t6gkb" podStartSLOduration=2.24562942 podStartE2EDuration="4.414560195s" podCreationTimestamp="2026-05-26 11:27:36 +0000 UTC" firstStartedPulling="2026-05-26 11:27:37.250789089 +0000 UTC m=+179.073173292" lastFinishedPulling="2026-05-26 11:27:39.419719864 +0000 UTC m=+181.242104067" observedRunningTime="2026-05-26 11:27:40.413814436 +0000 UTC m=+182.236198673" watchObservedRunningTime="2026-05-26 11:27:40.414560195 +0000 UTC m=+182.236944419"
May 26 11:27:52.966372 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:52.966335    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" podUID="3f98f24f-77ce-4dd4-b266-efb841e0a16f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:27:57.671437 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.671408    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pkwcq"]
May 26 11:27:57.674430 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.674410    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.682287 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.682271    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 11:27:57.682660 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.682643    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 11:27:57.685441 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.685415    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 11:27:57.685441 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.685438    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 11:27:57.685612 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.685440    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gxzss\""
May 26 11:27:57.685612 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.685440    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 11:27:57.685612 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.685461    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 11:27:57.847420 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847389    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847558 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-wtmp\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847558 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847481    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-textfile\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847558 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847507    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tsg\" (UniqueName: \"kubernetes.io/projected/76681724-2bcb-45ed-9408-973e8969c50a-kube-api-access-s4tsg\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847558 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847535    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-sys\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847760 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847571    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-tls\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847760 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847614    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76681724-2bcb-45ed-9408-973e8969c50a-metrics-client-ca\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847760 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-root\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.847760 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.847685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948627 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948571    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-root\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948627 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948602    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948783 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948636    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948783 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948662    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-wtmp\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948783 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948686    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-root\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948783 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948696    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-textfile\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948783 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948746    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tsg\" (UniqueName: \"kubernetes.io/projected/76681724-2bcb-45ed-9408-973e8969c50a-kube-api-access-s4tsg\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.948783 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948782    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-sys\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.949107 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948810    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-tls\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.949107 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948844    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-wtmp\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.949107 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948858    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76681724-2bcb-45ed-9408-973e8969c50a-metrics-client-ca\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.949107 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.948888    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76681724-2bcb-45ed-9408-973e8969c50a-sys\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.949107 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.949022    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-textfile\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.949362 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.949221    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.949415 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.949361    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76681724-2bcb-45ed-9408-973e8969c50a-metrics-client-ca\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.950784 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.950768    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.951011 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.950992    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76681724-2bcb-45ed-9408-973e8969c50a-node-exporter-tls\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.958473 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.958455    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tsg\" (UniqueName: \"kubernetes.io/projected/76681724-2bcb-45ed-9408-973e8969c50a-kube-api-access-s4tsg\") pod \"node-exporter-pkwcq\" (UID: \"76681724-2bcb-45ed-9408-973e8969c50a\") " pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.983050 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:57.983030    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pkwcq"
May 26 11:27:57.990342 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:27:57.990312    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76681724_2bcb_45ed_9408_973e8969c50a.slice/crio-3cc8d24ccf4137d0be5cb0abbafcf8e70fd99b6e8b7e5f77ca9c51dbe75883d1 WatchSource:0}: Error finding container 3cc8d24ccf4137d0be5cb0abbafcf8e70fd99b6e8b7e5f77ca9c51dbe75883d1: Status 404 returned error can't find the container with id 3cc8d24ccf4137d0be5cb0abbafcf8e70fd99b6e8b7e5f77ca9c51dbe75883d1
May 26 11:27:58.416781 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:58.416754    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkwcq" event={"ID":"76681724-2bcb-45ed-9408-973e8969c50a","Type":"ContainerStarted","Data":"3cc8d24ccf4137d0be5cb0abbafcf8e70fd99b6e8b7e5f77ca9c51dbe75883d1"}
May 26 11:27:58.534750 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:58.534717    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f654b6cd-s6jmw"]
May 26 11:27:58.534887 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:27:58.534868    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-f654b6cd-s6jmw" podUID="e6d55b72-2856-4c8a-b33d-04403fffd805"
May 26 11:27:59.420379 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.420346    2557 generic.go:358] "Generic (PLEG): container finished" podID="76681724-2bcb-45ed-9408-973e8969c50a" containerID="d85efac1f2a30354f457b3f869ad8b41fac9705a7f9cb6bfb7f38067ec4ea96c" exitCode=0
May 26 11:27:59.420697 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.420425    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkwcq" event={"ID":"76681724-2bcb-45ed-9408-973e8969c50a","Type":"ContainerDied","Data":"d85efac1f2a30354f457b3f869ad8b41fac9705a7f9cb6bfb7f38067ec4ea96c"}
May 26 11:27:59.420697 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.420516    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:27:59.428203 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.428185    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:27:59.561944 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.561920    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6d55b72-2856-4c8a-b33d-04403fffd805-ca-trust-extracted\") pod \"e6d55b72-2856-4c8a-b33d-04403fffd805\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") "
May 26 11:27:59.562113 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.561957    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-bound-sa-token\") pod \"e6d55b72-2856-4c8a-b33d-04403fffd805\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") "
May 26 11:27:59.562113 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.561997    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qqtx\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-kube-api-access-8qqtx\") pod \"e6d55b72-2856-4c8a-b33d-04403fffd805\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") "
May 26 11:27:59.562113 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562035    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-certificates\") pod \"e6d55b72-2856-4c8a-b33d-04403fffd805\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") "
May 26 11:27:59.562113 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562086    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-image-registry-private-configuration\") pod \"e6d55b72-2856-4c8a-b33d-04403fffd805\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") "
May 26 11:27:59.562290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562207    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d55b72-2856-4c8a-b33d-04403fffd805-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e6d55b72-2856-4c8a-b33d-04403fffd805" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:27:59.562290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562222    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-installation-pull-secrets\") pod \"e6d55b72-2856-4c8a-b33d-04403fffd805\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") "
May 26 11:27:59.562290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562278    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-trusted-ca\") pod \"e6d55b72-2856-4c8a-b33d-04403fffd805\" (UID: \"e6d55b72-2856-4c8a-b33d-04403fffd805\") "
May 26 11:27:59.562437 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562383    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e6d55b72-2856-4c8a-b33d-04403fffd805" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:59.562976 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562707    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e6d55b72-2856-4c8a-b33d-04403fffd805" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:27:59.562976 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562902    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-trusted-ca\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:27:59.562976 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562922    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6d55b72-2856-4c8a-b33d-04403fffd805-ca-trust-extracted\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:27:59.562976 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.562937    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-certificates\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:27:59.564338 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.564313    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e6d55b72-2856-4c8a-b33d-04403fffd805" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:27:59.564416 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.564369    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-kube-api-access-8qqtx" (OuterVolumeSpecName: "kube-api-access-8qqtx") pod "e6d55b72-2856-4c8a-b33d-04403fffd805" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805"). InnerVolumeSpecName "kube-api-access-8qqtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:27:59.564537 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.564521    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e6d55b72-2856-4c8a-b33d-04403fffd805" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:59.564598 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.564582    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e6d55b72-2856-4c8a-b33d-04403fffd805" (UID: "e6d55b72-2856-4c8a-b33d-04403fffd805"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:27:59.663417 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.663395    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-bound-sa-token\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:27:59.663417 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.663415    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qqtx\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-kube-api-access-8qqtx\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:27:59.663555 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.663426    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-image-registry-private-configuration\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:27:59.663555 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:27:59.663435    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6d55b72-2856-4c8a-b33d-04403fffd805-installation-pull-secrets\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:28:00.424722 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.424694    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f654b6cd-s6jmw"
May 26 11:28:00.424722 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.424702    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkwcq" event={"ID":"76681724-2bcb-45ed-9408-973e8969c50a","Type":"ContainerStarted","Data":"4b784ffd56b5a2bca6155ec340c6be7989b2a0bca9f7c8417a452864390b2b6d"}
May 26 11:28:00.425222 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.424736    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkwcq" event={"ID":"76681724-2bcb-45ed-9408-973e8969c50a","Type":"ContainerStarted","Data":"9535dfab91ba0a83651d08b3cef17b6c326f7632d3ec9d391d8c2cd8ad7aaba6"}
May 26 11:28:00.449404 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.449347    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pkwcq" podStartSLOduration=2.463086845 podStartE2EDuration="3.449337907s" podCreationTimestamp="2026-05-26 11:27:57 +0000 UTC" firstStartedPulling="2026-05-26 11:27:57.992258571 +0000 UTC m=+199.814642787" lastFinishedPulling="2026-05-26 11:27:58.978509646 +0000 UTC m=+200.800893849" observedRunningTime="2026-05-26 11:28:00.448548377 +0000 UTC m=+202.270932601" watchObservedRunningTime="2026-05-26 11:28:00.449337907 +0000 UTC m=+202.271722131"
May 26 11:28:00.489671 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.489650    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f654b6cd-s6jmw"]
May 26 11:28:00.498352 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.498331    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f654b6cd-s6jmw"]
May 26 11:28:00.671123 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.671096    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6d55b72-2856-4c8a-b33d-04403fffd805-registry-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:28:00.798194 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:00.798170    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6d55b72-2856-4c8a-b33d-04403fffd805" path="/var/lib/kubelet/pods/e6d55b72-2856-4c8a-b33d-04403fffd805/volumes"
May 26 11:28:02.966363 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:02.966327    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" podUID="3f98f24f-77ce-4dd4-b266-efb841e0a16f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:28:12.966527 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:12.966488    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" podUID="3f98f24f-77ce-4dd4-b266-efb841e0a16f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:28:12.966980 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:12.966569    2557 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq"
May 26 11:28:12.967180 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:12.967145    2557 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6af9745ab05b72063a1493121e7b8775cd1553a01682f13b307d5b2dc8902a71"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 26 11:28:12.967264 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:12.967220    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" podUID="3f98f24f-77ce-4dd4-b266-efb841e0a16f" containerName="service-proxy" containerID="cri-o://6af9745ab05b72063a1493121e7b8775cd1553a01682f13b307d5b2dc8902a71" gracePeriod=30
May 26 11:28:13.464200 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:13.464169    2557 generic.go:358] "Generic (PLEG): container finished" podID="3f98f24f-77ce-4dd4-b266-efb841e0a16f" containerID="6af9745ab05b72063a1493121e7b8775cd1553a01682f13b307d5b2dc8902a71" exitCode=2
May 26 11:28:13.464346 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:13.464225    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" event={"ID":"3f98f24f-77ce-4dd4-b266-efb841e0a16f","Type":"ContainerDied","Data":"6af9745ab05b72063a1493121e7b8775cd1553a01682f13b307d5b2dc8902a71"}
May 26 11:28:13.464346 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:13.464248    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc47479fd-ftbsq" event={"ID":"3f98f24f-77ce-4dd4-b266-efb841e0a16f","Type":"ContainerStarted","Data":"39b3a18cc33aba68431149ed933137eb24ec643e62723b380c26fa17bdd803c8"}
May 26 11:28:50.639664 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:50.639628    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:28:50.641811 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:50.641785    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20-metrics-certs\") pod \"network-metrics-daemon-t24mf\" (UID: \"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20\") " pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:28:50.798516 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:50.798488    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-75s55\""
May 26 11:28:50.806030 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:50.806012    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t24mf"
May 26 11:28:50.922301 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:50.922241    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t24mf"]
May 26 11:28:50.925141 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:28:50.925060    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e2271b_ccaf_4a22_a1ba_8bc9f6479b20.slice/crio-6c58ccd146350985c2938acc24f352787a9450cf7c63fe94019721258af44a80 WatchSource:0}: Error finding container 6c58ccd146350985c2938acc24f352787a9450cf7c63fe94019721258af44a80: Status 404 returned error can't find the container with id 6c58ccd146350985c2938acc24f352787a9450cf7c63fe94019721258af44a80
May 26 11:28:51.555543 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:51.555507    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t24mf" event={"ID":"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20","Type":"ContainerStarted","Data":"6c58ccd146350985c2938acc24f352787a9450cf7c63fe94019721258af44a80"}
May 26 11:28:52.559781 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:52.559744    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t24mf" event={"ID":"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20","Type":"ContainerStarted","Data":"dad499fb8faf88ed30477f8a9844e66e3280d896b5c5b445a1d9dd1329608cd3"}
May 26 11:28:52.559781 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:52.559784    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t24mf" event={"ID":"d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20","Type":"ContainerStarted","Data":"aa4c4d42c790b0533aa81f4f6396c4910cea8733d19c12f7a59a98093d9ac10b"}
May 26 11:28:52.578940 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:28:52.578897    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t24mf" podStartSLOduration=253.704041862 podStartE2EDuration="4m14.578883031s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="2026-05-26 11:28:50.926830457 +0000 UTC m=+252.749214660" lastFinishedPulling="2026-05-26 11:28:51.801671623 +0000 UTC m=+253.624055829" observedRunningTime="2026-05-26 11:28:52.577876232 +0000 UTC m=+254.400260456" watchObservedRunningTime="2026-05-26 11:28:52.578883031 +0000 UTC m=+254.401267256"
May 26 11:29:19.305408 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:29:19.305374    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9fbqd" podUID="89d04d62-aed1-4b37-93fe-170fafdb6e2d"
May 26 11:29:19.305408 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:29:19.305374    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9" podUID="61089709-9697-43cb-bf80-a0d676d4511c"
May 26 11:29:19.620306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:19.620241    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fbqd"
May 26 11:29:19.620430 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:19.620239    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:29:22.661445 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.661414    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:29:22.663726 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.663706    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/61089709-9697-43cb-bf80-a0d676d4511c-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-845q9\" (UID: \"61089709-9697-43cb-bf80-a0d676d4511c\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:29:22.761887 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.761864    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:29:22.761987 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.761905    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:29:22.764069 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.764034    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89d04d62-aed1-4b37-93fe-170fafdb6e2d-metrics-tls\") pod \"dns-default-9fbqd\" (UID: \"89d04d62-aed1-4b37-93fe-170fafdb6e2d\") " pod="openshift-dns/dns-default-9fbqd"
May 26 11:29:22.764069 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.764041    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9e2f963-efe8-4935-987d-0661094ac33f-cert\") pod \"ingress-canary-9v24s\" (UID: \"f9e2f963-efe8-4935-987d-0661094ac33f\") " pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:29:22.798150 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.798126    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zx2q6\""
May 26 11:29:22.805793 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.805776    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9v24s"
May 26 11:29:22.924450 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.924378    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mlcjf\""
May 26 11:29:22.924610 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.924591    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9pgx\""
May 26 11:29:22.924929 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.924909    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9v24s"]
May 26 11:29:22.927601 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:29:22.927579    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e2f963_efe8_4935_987d_0661094ac33f.slice/crio-a2bc11f440f46374f1705f373cc8a88e4c91878b3e5916d48bca3401133c989f WatchSource:0}: Error finding container a2bc11f440f46374f1705f373cc8a88e4c91878b3e5916d48bca3401133c989f: Status 404 returned error can't find the container with id a2bc11f440f46374f1705f373cc8a88e4c91878b3e5916d48bca3401133c989f
May 26 11:29:22.931662 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.931642    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"
May 26 11:29:22.931728 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:22.931668    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fbqd"
May 26 11:29:23.056852 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:23.056821    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-845q9"]
May 26 11:29:23.060508 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:29:23.060485    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61089709_9697_43cb_bf80_a0d676d4511c.slice/crio-c88d60c46aa2ea8af52e120ffc7eac81a17536c676136b56198dfc2dc05be580 WatchSource:0}: Error finding container c88d60c46aa2ea8af52e120ffc7eac81a17536c676136b56198dfc2dc05be580: Status 404 returned error can't find the container with id c88d60c46aa2ea8af52e120ffc7eac81a17536c676136b56198dfc2dc05be580
May 26 11:29:23.074140 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:23.074119    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fbqd"]
May 26 11:29:23.076751 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:29:23.076731    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d04d62_aed1_4b37_93fe_170fafdb6e2d.slice/crio-0fb97c97b657fce2d2d50331a4b624057af04a464fb9fafbbb6d2d328b04c87f WatchSource:0}: Error finding container 0fb97c97b657fce2d2d50331a4b624057af04a464fb9fafbbb6d2d328b04c87f: Status 404 returned error can't find the container with id 0fb97c97b657fce2d2d50331a4b624057af04a464fb9fafbbb6d2d328b04c87f
May 26 11:29:23.633379 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:23.633315    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9" event={"ID":"61089709-9697-43cb-bf80-a0d676d4511c","Type":"ContainerStarted","Data":"c88d60c46aa2ea8af52e120ffc7eac81a17536c676136b56198dfc2dc05be580"}
May 26 11:29:23.634680 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:23.634634    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9v24s" event={"ID":"f9e2f963-efe8-4935-987d-0661094ac33f","Type":"ContainerStarted","Data":"a2bc11f440f46374f1705f373cc8a88e4c91878b3e5916d48bca3401133c989f"}
May 26 11:29:23.636010 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:23.635978    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fbqd" event={"ID":"89d04d62-aed1-4b37-93fe-170fafdb6e2d","Type":"ContainerStarted","Data":"0fb97c97b657fce2d2d50331a4b624057af04a464fb9fafbbb6d2d328b04c87f"}
May 26 11:29:25.642629 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.642590    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9v24s" event={"ID":"f9e2f963-efe8-4935-987d-0661094ac33f","Type":"ContainerStarted","Data":"7e1c6be5b684a244eda33d2c3a0fe95bd8ff9c5816b3d236b62fddfd23838f56"}
May 26 11:29:25.644127 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.644095    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fbqd" event={"ID":"89d04d62-aed1-4b37-93fe-170fafdb6e2d","Type":"ContainerStarted","Data":"71d7db4ffd3aae3cc474983b2350bb2a78e5f69e5ae0916748fbfac6a3a67589"}
May 26 11:29:25.644259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.644132    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fbqd" event={"ID":"89d04d62-aed1-4b37-93fe-170fafdb6e2d","Type":"ContainerStarted","Data":"4eef65f8a0674f1b9c6603063a7875c733eea3cc5b15e5b18cebce880feec1ac"}
May 26 11:29:25.644259 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.644221    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9fbqd"
May 26 11:29:25.645288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.645270    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9" event={"ID":"61089709-9697-43cb-bf80-a0d676d4511c","Type":"ContainerStarted","Data":"7797d9d3eb7709ba06d014cb611c9069c84b6b2a08dcf921c3e5d270881b9014"}
May 26 11:29:25.662410 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.662373    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9v24s" podStartSLOduration=251.54885677 podStartE2EDuration="4m13.662363795s" podCreationTimestamp="2026-05-26 11:25:12 +0000 UTC" firstStartedPulling="2026-05-26 11:29:22.929390724 +0000 UTC m=+284.751774927" lastFinishedPulling="2026-05-26 11:29:25.042897743 +0000 UTC m=+286.865281952" observedRunningTime="2026-05-26 11:29:25.661864947 +0000 UTC m=+287.484249169" watchObservedRunningTime="2026-05-26 11:29:25.662363795 +0000 UTC m=+287.484748042"
May 26 11:29:25.687102 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.687035    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-845q9" podStartSLOduration=266.711284524 podStartE2EDuration="4m28.687020947s" podCreationTimestamp="2026-05-26 11:24:57 +0000 UTC" firstStartedPulling="2026-05-26 11:29:23.062356812 +0000 UTC m=+284.884741018" lastFinishedPulling="2026-05-26 11:29:25.038093225 +0000 UTC m=+286.860477441" observedRunningTime="2026-05-26 11:29:25.685264503 +0000 UTC m=+287.507648729" watchObservedRunningTime="2026-05-26 11:29:25.687020947 +0000 UTC m=+287.509405173"
May 26 11:29:25.705533 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:25.705495    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-9fbqd" podStartSLOduration=251.748088724 podStartE2EDuration="4m13.705483409s" podCreationTimestamp="2026-05-26 11:25:12 +0000 UTC" firstStartedPulling="2026-05-26 11:29:23.080689035 +0000 UTC m=+284.903073238" lastFinishedPulling="2026-05-26 11:29:25.038083703 +0000 UTC m=+286.860467923" observedRunningTime="2026-05-26 11:29:25.704290335 +0000 UTC m=+287.526674554" watchObservedRunningTime="2026-05-26 11:29:25.705483409 +0000 UTC m=+287.527867634"
May 26 11:29:35.649668 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:35.649640    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9fbqd"
May 26 11:29:38.701815 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:38.701784    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:29:38.702240 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:38.701890    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:29:38.704763 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:29:38.704743    2557 kubelet.go:1628] "Image garbage collection succeeded"
May 26 11:33:24.672963 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.672927    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-x6rsx"]
May 26 11:33:24.675819 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.675803    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:24.678639 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.678619    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 11:33:24.678756 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.678717    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bsnvz\""
May 26 11:33:24.678756 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.678733    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 11:33:24.679765 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.679752    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 11:33:24.686827 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.686805    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-x6rsx"]
May 26 11:33:24.765406 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.765379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zcc\" (UniqueName: \"kubernetes.io/projected/ab42601b-d8ff-4e76-afb7-7029cfabac7d-kube-api-access-89zcc\") pod \"seaweedfs-79f457656b-x6rsx\" (UID: \"ab42601b-d8ff-4e76-afb7-7029cfabac7d\") " pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:24.765509 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.765414    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ab42601b-d8ff-4e76-afb7-7029cfabac7d-data\") pod \"seaweedfs-79f457656b-x6rsx\" (UID: \"ab42601b-d8ff-4e76-afb7-7029cfabac7d\") " pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:24.866686 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.866661    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89zcc\" (UniqueName: \"kubernetes.io/projected/ab42601b-d8ff-4e76-afb7-7029cfabac7d-kube-api-access-89zcc\") pod \"seaweedfs-79f457656b-x6rsx\" (UID: \"ab42601b-d8ff-4e76-afb7-7029cfabac7d\") " pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:24.866768 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.866695    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ab42601b-d8ff-4e76-afb7-7029cfabac7d-data\") pod \"seaweedfs-79f457656b-x6rsx\" (UID: \"ab42601b-d8ff-4e76-afb7-7029cfabac7d\") " pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:24.866984 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.866970    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ab42601b-d8ff-4e76-afb7-7029cfabac7d-data\") pod \"seaweedfs-79f457656b-x6rsx\" (UID: \"ab42601b-d8ff-4e76-afb7-7029cfabac7d\") " pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:24.877856 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.877835    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zcc\" (UniqueName: \"kubernetes.io/projected/ab42601b-d8ff-4e76-afb7-7029cfabac7d-kube-api-access-89zcc\") pod \"seaweedfs-79f457656b-x6rsx\" (UID: \"ab42601b-d8ff-4e76-afb7-7029cfabac7d\") " pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:24.985736 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:24.985685    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:25.106269 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:25.106244    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-x6rsx"]
May 26 11:33:25.109120 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:33:25.109089    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab42601b_d8ff_4e76_afb7_7029cfabac7d.slice/crio-b762f26031a14a103fd7103a9bf0ccd086d7a8cd99ddd8470da01755962ac351 WatchSource:0}: Error finding container b762f26031a14a103fd7103a9bf0ccd086d7a8cd99ddd8470da01755962ac351: Status 404 returned error can't find the container with id b762f26031a14a103fd7103a9bf0ccd086d7a8cd99ddd8470da01755962ac351
May 26 11:33:25.110295 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:25.110280    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:33:25.236313 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:25.236252    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-x6rsx" event={"ID":"ab42601b-d8ff-4e76-afb7-7029cfabac7d","Type":"ContainerStarted","Data":"b762f26031a14a103fd7103a9bf0ccd086d7a8cd99ddd8470da01755962ac351"}
May 26 11:33:29.248769 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:29.248733    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-x6rsx" event={"ID":"ab42601b-d8ff-4e76-afb7-7029cfabac7d","Type":"ContainerStarted","Data":"58195dab0463ae75909c06f16c1073422d436b261b80e4b0bd80f23a339c2373"}
May 26 11:33:29.249232 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:29.248786    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:33:29.264896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:29.264839    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-x6rsx" podStartSLOduration=1.645366953 podStartE2EDuration="5.264825975s" podCreationTimestamp="2026-05-26 11:33:24 +0000 UTC" firstStartedPulling="2026-05-26 11:33:25.110404727 +0000 UTC m=+526.932788936" lastFinishedPulling="2026-05-26 11:33:28.729863749 +0000 UTC m=+530.552247958" observedRunningTime="2026-05-26 11:33:29.264717321 +0000 UTC m=+531.087101546" watchObservedRunningTime="2026-05-26 11:33:29.264825975 +0000 UTC m=+531.087210200"
May 26 11:33:35.253405 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:33:35.253376    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-x6rsx"
May 26 11:34:36.557831 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.557796    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-w4h84"]
May 26 11:34:36.559625 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.559608    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.563547 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.563528    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 26 11:34:36.563677 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.563548    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kggmw\""
May 26 11:34:36.573683 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.573664    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-w4h84"]
May 26 11:34:36.584997 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.584975    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-2mh8b"]
May 26 11:34:36.586781 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.586768    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:36.589746 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.589731    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 26 11:34:36.590026 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.590014    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-8tlnw\""
May 26 11:34:36.598909 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.598889    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-2mh8b"]
May 26 11:34:36.646657 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.646632    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4l5h\" (UniqueName: \"kubernetes.io/projected/0c66c557-047c-4540-b226-6b0831e097f2-kube-api-access-g4l5h\") pod \"model-serving-api-86f7b4b499-w4h84\" (UID: \"0c66c557-047c-4540-b226-6b0831e097f2\") " pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.646742 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.646669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8f40655-494c-44f3-9359-5cbbb77ddbcb-cert\") pod \"odh-model-controller-696fc77849-2mh8b\" (UID: \"f8f40655-494c-44f3-9359-5cbbb77ddbcb\") " pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:36.646742 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.646686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c66c557-047c-4540-b226-6b0831e097f2-tls-certs\") pod \"model-serving-api-86f7b4b499-w4h84\" (UID: \"0c66c557-047c-4540-b226-6b0831e097f2\") " pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.646742 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.646720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmd5\" (UniqueName: \"kubernetes.io/projected/f8f40655-494c-44f3-9359-5cbbb77ddbcb-kube-api-access-8gmd5\") pod \"odh-model-controller-696fc77849-2mh8b\" (UID: \"f8f40655-494c-44f3-9359-5cbbb77ddbcb\") " pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:36.747919 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.747896    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g4l5h\" (UniqueName: \"kubernetes.io/projected/0c66c557-047c-4540-b226-6b0831e097f2-kube-api-access-g4l5h\") pod \"model-serving-api-86f7b4b499-w4h84\" (UID: \"0c66c557-047c-4540-b226-6b0831e097f2\") " pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.748022 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.747942    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8f40655-494c-44f3-9359-5cbbb77ddbcb-cert\") pod \"odh-model-controller-696fc77849-2mh8b\" (UID: \"f8f40655-494c-44f3-9359-5cbbb77ddbcb\") " pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:36.748022 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.747971    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c66c557-047c-4540-b226-6b0831e097f2-tls-certs\") pod \"model-serving-api-86f7b4b499-w4h84\" (UID: \"0c66c557-047c-4540-b226-6b0831e097f2\") " pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.748022 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.748001    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmd5\" (UniqueName: \"kubernetes.io/projected/f8f40655-494c-44f3-9359-5cbbb77ddbcb-kube-api-access-8gmd5\") pod \"odh-model-controller-696fc77849-2mh8b\" (UID: \"f8f40655-494c-44f3-9359-5cbbb77ddbcb\") " pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:36.748208 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:34:36.748040    2557 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 26 11:34:36.748208 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:34:36.748120    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f40655-494c-44f3-9359-5cbbb77ddbcb-cert podName:f8f40655-494c-44f3-9359-5cbbb77ddbcb nodeName:}" failed. No retries permitted until 2026-05-26 11:34:37.248102087 +0000 UTC m=+599.070486291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f8f40655-494c-44f3-9359-5cbbb77ddbcb-cert") pod "odh-model-controller-696fc77849-2mh8b" (UID: "f8f40655-494c-44f3-9359-5cbbb77ddbcb") : secret "odh-model-controller-webhook-cert" not found
May 26 11:34:36.750222 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.750195    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c66c557-047c-4540-b226-6b0831e097f2-tls-certs\") pod \"model-serving-api-86f7b4b499-w4h84\" (UID: \"0c66c557-047c-4540-b226-6b0831e097f2\") " pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.757994 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.757971    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmd5\" (UniqueName: \"kubernetes.io/projected/f8f40655-494c-44f3-9359-5cbbb77ddbcb-kube-api-access-8gmd5\") pod \"odh-model-controller-696fc77849-2mh8b\" (UID: \"f8f40655-494c-44f3-9359-5cbbb77ddbcb\") " pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:36.758097 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.758006    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4l5h\" (UniqueName: \"kubernetes.io/projected/0c66c557-047c-4540-b226-6b0831e097f2-kube-api-access-g4l5h\") pod \"model-serving-api-86f7b4b499-w4h84\" (UID: \"0c66c557-047c-4540-b226-6b0831e097f2\") " pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.868660 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.868607    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:36.993142 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:36.993109    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-w4h84"]
May 26 11:34:36.995639 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:34:36.995607    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c66c557_047c_4540_b226_6b0831e097f2.slice/crio-8803c2c2d1aabfe0212347773101442179242ad7f1eff53fbb5d83c393c9af94 WatchSource:0}: Error finding container 8803c2c2d1aabfe0212347773101442179242ad7f1eff53fbb5d83c393c9af94: Status 404 returned error can't find the container with id 8803c2c2d1aabfe0212347773101442179242ad7f1eff53fbb5d83c393c9af94
May 26 11:34:37.252277 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:37.252209    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8f40655-494c-44f3-9359-5cbbb77ddbcb-cert\") pod \"odh-model-controller-696fc77849-2mh8b\" (UID: \"f8f40655-494c-44f3-9359-5cbbb77ddbcb\") " pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:37.254321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:37.254303    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8f40655-494c-44f3-9359-5cbbb77ddbcb-cert\") pod \"odh-model-controller-696fc77849-2mh8b\" (UID: \"f8f40655-494c-44f3-9359-5cbbb77ddbcb\") " pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:37.427641 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:37.427605    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-w4h84" event={"ID":"0c66c557-047c-4540-b226-6b0831e097f2","Type":"ContainerStarted","Data":"8803c2c2d1aabfe0212347773101442179242ad7f1eff53fbb5d83c393c9af94"}
May 26 11:34:37.495426 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:37.495395    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:37.624573 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:37.624544    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-2mh8b"]
May 26 11:34:37.627163 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:34:37.627132    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f40655_494c_44f3_9359_5cbbb77ddbcb.slice/crio-5a0ae9aa869e5e9c22e33eea67fbef7a978df4d98f00d045c7383edf0d28275a WatchSource:0}: Error finding container 5a0ae9aa869e5e9c22e33eea67fbef7a978df4d98f00d045c7383edf0d28275a: Status 404 returned error can't find the container with id 5a0ae9aa869e5e9c22e33eea67fbef7a978df4d98f00d045c7383edf0d28275a
May 26 11:34:38.432482 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:38.432445    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-2mh8b" event={"ID":"f8f40655-494c-44f3-9359-5cbbb77ddbcb","Type":"ContainerStarted","Data":"5a0ae9aa869e5e9c22e33eea67fbef7a978df4d98f00d045c7383edf0d28275a"}
May 26 11:34:40.593130 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:40.593101    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:34:40.593532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:40.593282    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:34:41.441946 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:41.441907    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-w4h84" event={"ID":"0c66c557-047c-4540-b226-6b0831e097f2","Type":"ContainerStarted","Data":"b313a7cc42ba52e89dd9a24a1506cea7cb4755bdd1b0d14cfbd9ff7a11550791"}
May 26 11:34:41.442173 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:41.442155    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:34:41.443144 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:41.443116    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-2mh8b" event={"ID":"f8f40655-494c-44f3-9359-5cbbb77ddbcb","Type":"ContainerStarted","Data":"d4f6f079c02a59b9a9ca33a7e83b215a4c5698081fca729399abd23543111044"}
May 26 11:34:41.443294 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:41.443278    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:41.462681 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:41.462645    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-w4h84" podStartSLOduration=1.8217995729999998 podStartE2EDuration="5.462633888s" podCreationTimestamp="2026-05-26 11:34:36 +0000 UTC" firstStartedPulling="2026-05-26 11:34:36.997086972 +0000 UTC m=+598.819471189" lastFinishedPulling="2026-05-26 11:34:40.637921302 +0000 UTC m=+602.460305504" observedRunningTime="2026-05-26 11:34:41.461232204 +0000 UTC m=+603.283616429" watchObservedRunningTime="2026-05-26 11:34:41.462633888 +0000 UTC m=+603.285018168"
May 26 11:34:41.481057 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:41.481012    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-2mh8b" podStartSLOduration=2.472460532 podStartE2EDuration="5.480995614s" podCreationTimestamp="2026-05-26 11:34:36 +0000 UTC" firstStartedPulling="2026-05-26 11:34:37.628609207 +0000 UTC m=+599.450993417" lastFinishedPulling="2026-05-26 11:34:40.637144277 +0000 UTC m=+602.459528499" observedRunningTime="2026-05-26 11:34:41.47999009 +0000 UTC m=+603.302374316" watchObservedRunningTime="2026-05-26 11:34:41.480995614 +0000 UTC m=+603.303379840"
May 26 11:34:52.447164 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:52.447134    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-2mh8b"
May 26 11:34:52.449426 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:34:52.449404    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-w4h84"
May 26 11:35:14.404536 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.404503    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"]
May 26 11:35:14.475054 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.475024    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"]
May 26 11:35:14.475213 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.475175    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.483060 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.483035    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wtv9z\""
May 26 11:35:14.483060 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.483034    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-88958-predictor-serving-cert\""
May 26 11:35:14.483411 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.483391    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 11:35:14.483498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.483441    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\""
May 26 11:35:14.483556 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.483399    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 11:35:14.612856 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.612822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04fe5eed-cfb8-419d-a4c2-d27074305e69-isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.613010 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.612873    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe5eed-cfb8-419d-a4c2-d27074305e69-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.613010 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.612965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d89rt\" (UniqueName: \"kubernetes.io/projected/04fe5eed-cfb8-419d-a4c2-d27074305e69-kube-api-access-d89rt\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.613010 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.612997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04fe5eed-cfb8-419d-a4c2-d27074305e69-proxy-tls\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.713941 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.713873    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d89rt\" (UniqueName: \"kubernetes.io/projected/04fe5eed-cfb8-419d-a4c2-d27074305e69-kube-api-access-d89rt\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.713941 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.713909    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04fe5eed-cfb8-419d-a4c2-d27074305e69-proxy-tls\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.714175 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.714152    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04fe5eed-cfb8-419d-a4c2-d27074305e69-isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.714242 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.714204    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe5eed-cfb8-419d-a4c2-d27074305e69-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.714523 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.714504    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe5eed-cfb8-419d-a4c2-d27074305e69-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.714773 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.714752    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04fe5eed-cfb8-419d-a4c2-d27074305e69-isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.716245 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.716225    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04fe5eed-cfb8-419d-a4c2-d27074305e69-proxy-tls\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.724374 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.724354    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d89rt\" (UniqueName: \"kubernetes.io/projected/04fe5eed-cfb8-419d-a4c2-d27074305e69-kube-api-access-d89rt\") pod \"isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.784981 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.784950    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:14.922969 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:14.922941    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"]
May 26 11:35:14.926022 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:35:14.925992    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fe5eed_cfb8_419d_a4c2_d27074305e69.slice/crio-22431c102f620fecf119d7c11aaaa102f817d04df941aa9260115952ab317895 WatchSource:0}: Error finding container 22431c102f620fecf119d7c11aaaa102f817d04df941aa9260115952ab317895: Status 404 returned error can't find the container with id 22431c102f620fecf119d7c11aaaa102f817d04df941aa9260115952ab317895
May 26 11:35:15.531487 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:15.531435    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerStarted","Data":"22431c102f620fecf119d7c11aaaa102f817d04df941aa9260115952ab317895"}
May 26 11:35:19.545008 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:19.544976    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerStarted","Data":"25634136613f3866b703de549734fc0f919de3de68a4de2c0334fd1ffe3ad310"}
May 26 11:35:23.559706 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:23.559670    2557 generic.go:358] "Generic (PLEG): container finished" podID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerID="25634136613f3866b703de549734fc0f919de3de68a4de2c0334fd1ffe3ad310" exitCode=0
May 26 11:35:23.560137 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:23.559742    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerDied","Data":"25634136613f3866b703de549734fc0f919de3de68a4de2c0334fd1ffe3ad310"}
May 26 11:35:36.602948 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:36.602881    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerStarted","Data":"060bf8e350f3d0008c9fe172ff8bf67d67c171965986fbc627b15051d4f183f2"}
May 26 11:35:39.612662 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:39.612631    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerStarted","Data":"d095a3acb8320e8314952d2822283325c5a8694ee0e22cdcd135af22b18969d6"}
May 26 11:35:42.623396 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:42.623359    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerStarted","Data":"e482657511b039ebe6d9d60719d42e20647cabf4c999cd502189615083a77e9a"}
May 26 11:35:42.623760 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:42.623534    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:42.649709 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:42.649668    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podStartSLOduration=2.001013882 podStartE2EDuration="28.649655106s" podCreationTimestamp="2026-05-26 11:35:14 +0000 UTC" firstStartedPulling="2026-05-26 11:35:14.92785418 +0000 UTC m=+636.750238383" lastFinishedPulling="2026-05-26 11:35:41.576495398 +0000 UTC m=+663.398879607" observedRunningTime="2026-05-26 11:35:42.647587033 +0000 UTC m=+664.469971261" watchObservedRunningTime="2026-05-26 11:35:42.649655106 +0000 UTC m=+664.472039330"
May 26 11:35:43.625949 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:43.625911    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:43.626375 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:43.625990    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:43.627320 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:43.627287    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:35:43.627959 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:43.627927    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:35:43.631684 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:43.631665    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:35:44.628180 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:44.628140    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:35:44.628531 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:44.628410    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:35:45.630381 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:45.630342    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:35:45.630731 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:45.630552    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:35:55.630777 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:55.630727    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:35:55.631278 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:35:55.631124    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:36:05.630686 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:05.630630    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:36:05.631051 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:05.631029    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:36:15.630902 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:15.630850    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:36:15.631385 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:15.631361    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:36:25.630994 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:25.630950    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:36:25.631409 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:25.631390    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:36:35.630879 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:35.630789    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:36:35.631344 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:35.631246    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:36:45.630923 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:45.630885    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:36:45.631352 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:45.631118    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:36:59.493715 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.493678    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"]
May 26 11:36:59.494105 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.494039    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" containerID="cri-o://e482657511b039ebe6d9d60719d42e20647cabf4c999cd502189615083a77e9a" gracePeriod=30
May 26 11:36:59.494105 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.494030    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" containerID="cri-o://060bf8e350f3d0008c9fe172ff8bf67d67c171965986fbc627b15051d4f183f2" gracePeriod=30
May 26 11:36:59.494105 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.494048    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy" containerID="cri-o://d095a3acb8320e8314952d2822283325c5a8694ee0e22cdcd135af22b18969d6" gracePeriod=30
May 26 11:36:59.642174 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.642141    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"]
May 26 11:36:59.644314 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.644296    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.647408 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.647385    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-ea074-predictor-serving-cert\""
May 26 11:36:59.647522 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.647431    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\""
May 26 11:36:59.662053 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.661871    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0687fd1-5202-43b7-af26-951a9fdf49c2-isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.662053 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.661973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0687fd1-5202-43b7-af26-951a9fdf49c2-proxy-tls\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.668123 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.662514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0687fd1-5202-43b7-af26-951a9fdf49c2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.668123 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.662601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxwhc\" (UniqueName: \"kubernetes.io/projected/a0687fd1-5202-43b7-af26-951a9fdf49c2-kube-api-access-vxwhc\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.668123 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.665810    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"]
May 26 11:36:59.743822 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.743767    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"]
May 26 11:36:59.745877 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.745861    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.749087 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.749061    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-ea074-predictor-serving-cert\""
May 26 11:36:59.749183 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.749060    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\""
May 26 11:36:59.763019 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.762990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqs8x\" (UniqueName: \"kubernetes.io/projected/dc0cb65c-ebe3-4a03-a52a-677266309387-kube-api-access-pqs8x\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.763124 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763035    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0cb65c-ebe3-4a03-a52a-677266309387-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.763124 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763099    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0687fd1-5202-43b7-af26-951a9fdf49c2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.763211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.763211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763170    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vxwhc\" (UniqueName: \"kubernetes.io/projected/a0687fd1-5202-43b7-af26-951a9fdf49c2-kube-api-access-vxwhc\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.763306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0cb65c-ebe3-4a03-a52a-677266309387-isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.763306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763259    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0687fd1-5202-43b7-af26-951a9fdf49c2-isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.763408 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763318    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0687fd1-5202-43b7-af26-951a9fdf49c2-proxy-tls\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.763587 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.763470    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0687fd1-5202-43b7-af26-951a9fdf49c2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.764116 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.764066    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0687fd1-5202-43b7-af26-951a9fdf49c2-isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.764416 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.764401    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"]
May 26 11:36:59.765915 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.765888    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0687fd1-5202-43b7-af26-951a9fdf49c2-proxy-tls\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.777041 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.777014    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxwhc\" (UniqueName: \"kubernetes.io/projected/a0687fd1-5202-43b7-af26-951a9fdf49c2-kube-api-access-vxwhc\") pod \"isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:36:59.824896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.824875    2557 generic.go:358] "Generic (PLEG): container finished" podID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerID="d095a3acb8320e8314952d2822283325c5a8694ee0e22cdcd135af22b18969d6" exitCode=2
May 26 11:36:59.824982 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.824946    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerDied","Data":"d095a3acb8320e8314952d2822283325c5a8694ee0e22cdcd135af22b18969d6"}
May 26 11:36:59.863909 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.863881    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pqs8x\" (UniqueName: \"kubernetes.io/projected/dc0cb65c-ebe3-4a03-a52a-677266309387-kube-api-access-pqs8x\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.863981 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.863913    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0cb65c-ebe3-4a03-a52a-677266309387-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.863981 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.863948    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.864062 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.863991    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0cb65c-ebe3-4a03-a52a-677266309387-isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.864155 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:36:59.864133    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-serving-cert: secret "isvc-xgboost-graph-raw-ea074-predictor-serving-cert" not found
May 26 11:36:59.864291 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:36:59.864200    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls podName:dc0cb65c-ebe3-4a03-a52a-677266309387 nodeName:}" failed. No retries permitted until 2026-05-26 11:37:00.364182372 +0000 UTC m=+742.186566579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls") pod "isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" (UID: "dc0cb65c-ebe3-4a03-a52a-677266309387") : secret "isvc-xgboost-graph-raw-ea074-predictor-serving-cert" not found
May 26 11:36:59.864359 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.864290    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0cb65c-ebe3-4a03-a52a-677266309387-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.864596 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.864578    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0cb65c-ebe3-4a03-a52a-677266309387-isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.876881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.876852    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqs8x\" (UniqueName: \"kubernetes.io/projected/dc0cb65c-ebe3-4a03-a52a-677266309387-kube-api-access-pqs8x\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:36:59.953975 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:36:59.953955    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:37:00.083673 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.083652    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"]
May 26 11:37:00.086120 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:37:00.086092    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0687fd1_5202_43b7_af26_951a9fdf49c2.slice/crio-af784010fa59cfd93f9f43c563572e7c9ad1cfdb24cfbabcd793140be704bce5 WatchSource:0}: Error finding container af784010fa59cfd93f9f43c563572e7c9ad1cfdb24cfbabcd793140be704bce5: Status 404 returned error can't find the container with id af784010fa59cfd93f9f43c563572e7c9ad1cfdb24cfbabcd793140be704bce5
May 26 11:37:00.367728 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.367695    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:37:00.370000 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.369979    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls\") pod \"isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:37:00.655689 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.655607    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:37:00.781134 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.781104    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"]
May 26 11:37:00.784373 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:37:00.784350    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0cb65c_ebe3_4a03_a52a_677266309387.slice/crio-0bdfaa4378fa9c96a8754872068377e687b49dd348edbdcdfd6490128f17e97f WatchSource:0}: Error finding container 0bdfaa4378fa9c96a8754872068377e687b49dd348edbdcdfd6490128f17e97f: Status 404 returned error can't find the container with id 0bdfaa4378fa9c96a8754872068377e687b49dd348edbdcdfd6490128f17e97f
May 26 11:37:00.828799 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.828769    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerStarted","Data":"0bdfaa4378fa9c96a8754872068377e687b49dd348edbdcdfd6490128f17e97f"}
May 26 11:37:00.830051 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.830028    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerStarted","Data":"df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505"}
May 26 11:37:00.830131 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:00.830058    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerStarted","Data":"af784010fa59cfd93f9f43c563572e7c9ad1cfdb24cfbabcd793140be704bce5"}
May 26 11:37:01.834406 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:01.834367    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerStarted","Data":"67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574"}
May 26 11:37:03.626613 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:03.626575    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:37:03.846125 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:03.842216    2557 generic.go:358] "Generic (PLEG): container finished" podID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerID="060bf8e350f3d0008c9fe172ff8bf67d67c171965986fbc627b15051d4f183f2" exitCode=0
May 26 11:37:03.846125 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:03.842366    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerDied","Data":"060bf8e350f3d0008c9fe172ff8bf67d67c171965986fbc627b15051d4f183f2"}
May 26 11:37:04.846311 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:04.846279    2557 generic.go:358] "Generic (PLEG): container finished" podID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerID="67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574" exitCode=0
May 26 11:37:04.846728 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:04.846341    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerDied","Data":"67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574"}
May 26 11:37:04.847671 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:04.847646    2557 generic.go:358] "Generic (PLEG): container finished" podID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerID="df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505" exitCode=0
May 26 11:37:04.847725 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:04.847696    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerDied","Data":"df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505"}
May 26 11:37:05.630375 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:05.630329    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:37:05.630780 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:05.630745    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:37:05.854330 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:05.854293    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerStarted","Data":"5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338"}
May 26 11:37:05.854740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:05.854343    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerStarted","Data":"fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a"}
May 26 11:37:05.854740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:05.854556    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:37:05.876534 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:05.876468    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podStartSLOduration=6.876449372 podStartE2EDuration="6.876449372s" podCreationTimestamp="2026-05-26 11:36:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:37:05.87537673 +0000 UTC m=+747.697760965" watchObservedRunningTime="2026-05-26 11:37:05.876449372 +0000 UTC m=+747.698833620"
May 26 11:37:06.857942 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:06.857884    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:37:06.859714 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:06.859452    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:37:07.861328 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:07.861284    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:37:08.626414 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:08.626371    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:37:12.865692 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:12.865665    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:37:12.866323 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:12.866285    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:37:13.626754 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:13.626715    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:37:13.626933 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:13.626857    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:37:15.630458 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:15.630413    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:37:15.630906 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:15.630845    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:37:18.626534 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:18.626487    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:37:22.867104 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:22.867058    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:37:23.627330 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:23.627284    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:37:24.914544 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:24.914510    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerStarted","Data":"8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e"}
May 26 11:37:24.914544 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:24.914551    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerStarted","Data":"5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20"}
May 26 11:37:24.915054 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:24.914771    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:37:24.936260 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:24.936214    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podStartSLOduration=6.404629294 podStartE2EDuration="25.936202957s" podCreationTimestamp="2026-05-26 11:36:59 +0000 UTC" firstStartedPulling="2026-05-26 11:37:04.847755421 +0000 UTC m=+746.670139625" lastFinishedPulling="2026-05-26 11:37:24.379329082 +0000 UTC m=+766.201713288" observedRunningTime="2026-05-26 11:37:24.935304011 +0000 UTC m=+766.757688236" watchObservedRunningTime="2026-05-26 11:37:24.936202957 +0000 UTC m=+766.758587181"
May 26 11:37:25.630376 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:25.630334    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 26 11:37:25.630565 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:25.630480    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:37:25.630677 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:25.630641    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:37:25.630757 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:25.630744    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:37:25.917567 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:25.917491    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:37:25.918700 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:25.918672    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:37:26.920095 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:26.920044    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:37:28.626617 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:28.626576    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 26 11:37:29.929992 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:29.929954    2557 generic.go:358] "Generic (PLEG): container finished" podID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerID="e482657511b039ebe6d9d60719d42e20647cabf4c999cd502189615083a77e9a" exitCode=0
May 26 11:37:29.930380 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:29.930025    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerDied","Data":"e482657511b039ebe6d9d60719d42e20647cabf4c999cd502189615083a77e9a"}
May 26 11:37:30.141699 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.141670    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:37:30.213572 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.213500    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe5eed-cfb8-419d-a4c2-d27074305e69-kserve-provision-location\") pod \"04fe5eed-cfb8-419d-a4c2-d27074305e69\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") "
May 26 11:37:30.213572 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.213548    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04fe5eed-cfb8-419d-a4c2-d27074305e69-proxy-tls\") pod \"04fe5eed-cfb8-419d-a4c2-d27074305e69\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") "
May 26 11:37:30.213782 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.213587    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04fe5eed-cfb8-419d-a4c2-d27074305e69-isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\") pod \"04fe5eed-cfb8-419d-a4c2-d27074305e69\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") "
May 26 11:37:30.213782 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.213614    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d89rt\" (UniqueName: \"kubernetes.io/projected/04fe5eed-cfb8-419d-a4c2-d27074305e69-kube-api-access-d89rt\") pod \"04fe5eed-cfb8-419d-a4c2-d27074305e69\" (UID: \"04fe5eed-cfb8-419d-a4c2-d27074305e69\") "
May 26 11:37:30.213896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.213790    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe5eed-cfb8-419d-a4c2-d27074305e69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04fe5eed-cfb8-419d-a4c2-d27074305e69" (UID: "04fe5eed-cfb8-419d-a4c2-d27074305e69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:37:30.213990 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.213969    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fe5eed-cfb8-419d-a4c2-d27074305e69-isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config") pod "04fe5eed-cfb8-419d-a4c2-d27074305e69" (UID: "04fe5eed-cfb8-419d-a4c2-d27074305e69"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:37:30.215840 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.215812    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fe5eed-cfb8-419d-a4c2-d27074305e69-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04fe5eed-cfb8-419d-a4c2-d27074305e69" (UID: "04fe5eed-cfb8-419d-a4c2-d27074305e69"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:37:30.215840 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.215817    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe5eed-cfb8-419d-a4c2-d27074305e69-kube-api-access-d89rt" (OuterVolumeSpecName: "kube-api-access-d89rt") pod "04fe5eed-cfb8-419d-a4c2-d27074305e69" (UID: "04fe5eed-cfb8-419d-a4c2-d27074305e69"). InnerVolumeSpecName "kube-api-access-d89rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:37:30.314745 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.314719    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d89rt\" (UniqueName: \"kubernetes.io/projected/04fe5eed-cfb8-419d-a4c2-d27074305e69-kube-api-access-d89rt\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:37:30.314745 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.314743    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe5eed-cfb8-419d-a4c2-d27074305e69-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:37:30.314898 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.314753    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04fe5eed-cfb8-419d-a4c2-d27074305e69-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:37:30.314898 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.314764    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04fe5eed-cfb8-419d-a4c2-d27074305e69-isvc-raw-sklearn-batcher-88958-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:37:30.934532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.934503    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8" event={"ID":"04fe5eed-cfb8-419d-a4c2-d27074305e69","Type":"ContainerDied","Data":"22431c102f620fecf119d7c11aaaa102f817d04df941aa9260115952ab317895"}
May 26 11:37:30.934928 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.934545    2557 scope.go:117] "RemoveContainer" containerID="e482657511b039ebe6d9d60719d42e20647cabf4c999cd502189615083a77e9a"
May 26 11:37:30.934928 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.934545    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"
May 26 11:37:30.941945 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.941929    2557 scope.go:117] "RemoveContainer" containerID="d095a3acb8320e8314952d2822283325c5a8694ee0e22cdcd135af22b18969d6"
May 26 11:37:30.948639 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.948623    2557 scope.go:117] "RemoveContainer" containerID="060bf8e350f3d0008c9fe172ff8bf67d67c171965986fbc627b15051d4f183f2"
May 26 11:37:30.955574 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.955556    2557 scope.go:117] "RemoveContainer" containerID="25634136613f3866b703de549734fc0f919de3de68a4de2c0334fd1ffe3ad310"
May 26 11:37:30.963330 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.963307    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"]
May 26 11:37:30.967842 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:30.967822    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-88958-predictor-84f8f466bf-vdbh8"]
May 26 11:37:31.925124 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:31.925097    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:37:31.925602 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:31.925571    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:37:32.802622 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:32.802589    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" path="/var/lib/kubelet/pods/04fe5eed-cfb8-419d-a4c2-d27074305e69/volumes"
May 26 11:37:32.866883 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:32.866852    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:37:41.926493 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:41.926456    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:37:42.866261 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:42.866225    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:37:51.926498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:51.926459    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:37:52.866480 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:37:52.866444    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:38:01.925776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:01.925691    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:38:02.866494 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:02.866459    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:38:11.926521 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:11.926481    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:38:12.867621 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:12.867594    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:38:21.926119 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:21.926088    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:38:39.798588 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.798559    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"]
May 26 11:38:39.798974 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.798822    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" containerID="cri-o://fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a" gracePeriod=30
May 26 11:38:39.798974 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.798868    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kube-rbac-proxy" containerID="cri-o://5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338" gracePeriod=30
May 26 11:38:39.863727 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.863698    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"]
May 26 11:38:39.863972 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.863959    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="storage-initializer"
May 26 11:38:39.864034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.863974    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="storage-initializer"
May 26 11:38:39.864034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.863986    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy"
May 26 11:38:39.864034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.863991    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy"
May 26 11:38:39.864034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.864000    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent"
May 26 11:38:39.864034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.864005    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent"
May 26 11:38:39.864034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.864016    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container"
May 26 11:38:39.864034 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.864021    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container"
May 26 11:38:39.864383 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.864064    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kserve-container"
May 26 11:38:39.864383 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.864090    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="kube-rbac-proxy"
May 26 11:38:39.864383 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.864100    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="04fe5eed-cfb8-419d-a4c2-d27074305e69" containerName="agent"
May 26 11:38:39.865842 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.865824    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:39.868584 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.868565    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\""
May 26 11:38:39.868678 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.868567    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-019fa-predictor-serving-cert\""
May 26 11:38:39.876868 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.876845    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"]
May 26 11:38:39.915991 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.915969    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8002f55-cba5-4399-8c87-c8b7a27561b1-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:39.916102 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.916013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8002f55-cba5-4399-8c87-c8b7a27561b1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:39.916150 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.916124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8002f55-cba5-4399-8c87-c8b7a27561b1-isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:39.916206 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.916176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzn7j\" (UniqueName: \"kubernetes.io/projected/f8002f55-cba5-4399-8c87-c8b7a27561b1-kube-api-access-rzn7j\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:39.991303 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.991276    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"]
May 26 11:38:39.991600 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.991563    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" containerID="cri-o://5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20" gracePeriod=30
May 26 11:38:39.991720 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:39.991613    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kube-rbac-proxy" containerID="cri-o://8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e" gracePeriod=30
May 26 11:38:40.016249 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.016222    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"]
May 26 11:38:40.016480 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.016460    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8002f55-cba5-4399-8c87-c8b7a27561b1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.016533 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.016518    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8002f55-cba5-4399-8c87-c8b7a27561b1-isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.016577 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.016562    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rzn7j\" (UniqueName: \"kubernetes.io/projected/f8002f55-cba5-4399-8c87-c8b7a27561b1-kube-api-access-rzn7j\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.016620 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.016596    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8002f55-cba5-4399-8c87-c8b7a27561b1-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.016929 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.016907    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8002f55-cba5-4399-8c87-c8b7a27561b1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.017222 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.017203    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8002f55-cba5-4399-8c87-c8b7a27561b1-isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.018513 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.018495    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.019040 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.019023    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8002f55-cba5-4399-8c87-c8b7a27561b1-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.020968 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.020950    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\""
May 26 11:38:40.021059 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.020982    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-019fa-predictor-serving-cert\""
May 26 11:38:40.024426 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.024407    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzn7j\" (UniqueName: \"kubernetes.io/projected/f8002f55-cba5-4399-8c87-c8b7a27561b1-kube-api-access-rzn7j\") pod \"isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.029817 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.029798    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"]
May 26 11:38:40.117483 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.117460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44559c0-b91b-4c6a-be3a-c48d06807825-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.117572 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.117489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44559c0-b91b-4c6a-be3a-c48d06807825-isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.117572 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.117521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44559c0-b91b-4c6a-be3a-c48d06807825-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.117572 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.117546    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfn4v\" (UniqueName: \"kubernetes.io/projected/d44559c0-b91b-4c6a-be3a-c48d06807825-kube-api-access-pfn4v\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.131070 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.131043    2557 generic.go:358] "Generic (PLEG): container finished" podID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerID="8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e" exitCode=2
May 26 11:38:40.131171 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.131104    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerDied","Data":"8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e"}
May 26 11:38:40.132815 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.132794    2557 generic.go:358] "Generic (PLEG): container finished" podID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerID="5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338" exitCode=2
May 26 11:38:40.132891 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.132853    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerDied","Data":"5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338"}
May 26 11:38:40.175122 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.175100    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:40.218896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.218060    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44559c0-b91b-4c6a-be3a-c48d06807825-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.218896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.218159    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pfn4v\" (UniqueName: \"kubernetes.io/projected/d44559c0-b91b-4c6a-be3a-c48d06807825-kube-api-access-pfn4v\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.218896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.218238    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44559c0-b91b-4c6a-be3a-c48d06807825-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.218896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.218266    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44559c0-b91b-4c6a-be3a-c48d06807825-isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.218896 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.218586    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44559c0-b91b-4c6a-be3a-c48d06807825-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.219639 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.219006    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44559c0-b91b-4c6a-be3a-c48d06807825-isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.222186 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.222160    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44559c0-b91b-4c6a-be3a-c48d06807825-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.230558 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.230537    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfn4v\" (UniqueName: \"kubernetes.io/projected/d44559c0-b91b-4c6a-be3a-c48d06807825-kube-api-access-pfn4v\") pod \"isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.307456 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.307434    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"]
May 26 11:38:40.309709 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:38:40.309686    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8002f55_cba5_4399_8c87_c8b7a27561b1.slice/crio-4dbd3939d8f63f48cc431affa66f7e81e41a19bfc27553ada7f43fdaee6b455f WatchSource:0}: Error finding container 4dbd3939d8f63f48cc431affa66f7e81e41a19bfc27553ada7f43fdaee6b455f: Status 404 returned error can't find the container with id 4dbd3939d8f63f48cc431affa66f7e81e41a19bfc27553ada7f43fdaee6b455f
May 26 11:38:40.311520 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.311498    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:38:40.339724 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.339701    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:40.464010 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:40.463975    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"]
May 26 11:38:40.467675 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:38:40.467652    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44559c0_b91b_4c6a_be3a_c48d06807825.slice/crio-d4a78799f87824148cc0caddbfa9f06779ca552e2acff8b2982adea7d0923758 WatchSource:0}: Error finding container d4a78799f87824148cc0caddbfa9f06779ca552e2acff8b2982adea7d0923758: Status 404 returned error can't find the container with id d4a78799f87824148cc0caddbfa9f06779ca552e2acff8b2982adea7d0923758
May 26 11:38:41.137086 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:41.137054    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerStarted","Data":"2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e"}
May 26 11:38:41.137483 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:41.137099    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerStarted","Data":"d4a78799f87824148cc0caddbfa9f06779ca552e2acff8b2982adea7d0923758"}
May 26 11:38:41.138426 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:41.138406    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerStarted","Data":"eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619"}
May 26 11:38:41.138481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:41.138432    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerStarted","Data":"4dbd3939d8f63f48cc431affa66f7e81e41a19bfc27553ada7f43fdaee6b455f"}
May 26 11:38:41.921161 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:41.921122    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
May 26 11:38:41.926462 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:41.926437    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
May 26 11:38:42.862421 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:42.862377    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
May 26 11:38:42.866737 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:42.866709    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
May 26 11:38:43.538999 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.538977    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:38:43.646351 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.646273    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls\") pod \"dc0cb65c-ebe3-4a03-a52a-677266309387\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") "
May 26 11:38:43.646351 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.646316    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqs8x\" (UniqueName: \"kubernetes.io/projected/dc0cb65c-ebe3-4a03-a52a-677266309387-kube-api-access-pqs8x\") pod \"dc0cb65c-ebe3-4a03-a52a-677266309387\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") "
May 26 11:38:43.646351 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.646337    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0cb65c-ebe3-4a03-a52a-677266309387-isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"dc0cb65c-ebe3-4a03-a52a-677266309387\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") "
May 26 11:38:43.646607 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.646378    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0cb65c-ebe3-4a03-a52a-677266309387-kserve-provision-location\") pod \"dc0cb65c-ebe3-4a03-a52a-677266309387\" (UID: \"dc0cb65c-ebe3-4a03-a52a-677266309387\") "
May 26 11:38:43.646645 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.646626    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0cb65c-ebe3-4a03-a52a-677266309387-isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config") pod "dc0cb65c-ebe3-4a03-a52a-677266309387" (UID: "dc0cb65c-ebe3-4a03-a52a-677266309387"). InnerVolumeSpecName "isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:38:43.646693 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.646674    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0cb65c-ebe3-4a03-a52a-677266309387-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc0cb65c-ebe3-4a03-a52a-677266309387" (UID: "dc0cb65c-ebe3-4a03-a52a-677266309387"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:38:43.648414 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.648391    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0cb65c-ebe3-4a03-a52a-677266309387-kube-api-access-pqs8x" (OuterVolumeSpecName: "kube-api-access-pqs8x") pod "dc0cb65c-ebe3-4a03-a52a-677266309387" (UID: "dc0cb65c-ebe3-4a03-a52a-677266309387"). InnerVolumeSpecName "kube-api-access-pqs8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:38:43.648477 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.648436    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc0cb65c-ebe3-4a03-a52a-677266309387" (UID: "dc0cb65c-ebe3-4a03-a52a-677266309387"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:38:43.747049 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.747021    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc0cb65c-ebe3-4a03-a52a-677266309387-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:43.747049 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.747046    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqs8x\" (UniqueName: \"kubernetes.io/projected/dc0cb65c-ebe3-4a03-a52a-677266309387-kube-api-access-pqs8x\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:43.747213 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.747056    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc0cb65c-ebe3-4a03-a52a-677266309387-isvc-xgboost-graph-raw-ea074-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:43.747213 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.747065    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc0cb65c-ebe3-4a03-a52a-677266309387-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:43.934749 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:43.934729    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:38:44.048095 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048047    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0687fd1-5202-43b7-af26-951a9fdf49c2-kserve-provision-location\") pod \"a0687fd1-5202-43b7-af26-951a9fdf49c2\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") "
May 26 11:38:44.048229 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048126    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxwhc\" (UniqueName: \"kubernetes.io/projected/a0687fd1-5202-43b7-af26-951a9fdf49c2-kube-api-access-vxwhc\") pod \"a0687fd1-5202-43b7-af26-951a9fdf49c2\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") "
May 26 11:38:44.048229 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048177    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0687fd1-5202-43b7-af26-951a9fdf49c2-isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\") pod \"a0687fd1-5202-43b7-af26-951a9fdf49c2\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") "
May 26 11:38:44.048343 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048303    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0687fd1-5202-43b7-af26-951a9fdf49c2-proxy-tls\") pod \"a0687fd1-5202-43b7-af26-951a9fdf49c2\" (UID: \"a0687fd1-5202-43b7-af26-951a9fdf49c2\") "
May 26 11:38:44.048448 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048416    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0687fd1-5202-43b7-af26-951a9fdf49c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0687fd1-5202-43b7-af26-951a9fdf49c2" (UID: "a0687fd1-5202-43b7-af26-951a9fdf49c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:38:44.048530 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048488    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0687fd1-5202-43b7-af26-951a9fdf49c2-isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config") pod "a0687fd1-5202-43b7-af26-951a9fdf49c2" (UID: "a0687fd1-5202-43b7-af26-951a9fdf49c2"). InnerVolumeSpecName "isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:38:44.048580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048554    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0687fd1-5202-43b7-af26-951a9fdf49c2-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:44.048580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.048565    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0687fd1-5202-43b7-af26-951a9fdf49c2-isvc-sklearn-graph-raw-ea074-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:44.049945 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.049926    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0687fd1-5202-43b7-af26-951a9fdf49c2-kube-api-access-vxwhc" (OuterVolumeSpecName: "kube-api-access-vxwhc") pod "a0687fd1-5202-43b7-af26-951a9fdf49c2" (UID: "a0687fd1-5202-43b7-af26-951a9fdf49c2"). InnerVolumeSpecName "kube-api-access-vxwhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:38:44.050090 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.050048    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0687fd1-5202-43b7-af26-951a9fdf49c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0687fd1-5202-43b7-af26-951a9fdf49c2" (UID: "a0687fd1-5202-43b7-af26-951a9fdf49c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:38:44.148191 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.148163    2557 generic.go:358] "Generic (PLEG): container finished" podID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerID="fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a" exitCode=0
May 26 11:38:44.148306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.148243    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"
May 26 11:38:44.148306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.148242    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerDied","Data":"fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a"}
May 26 11:38:44.148306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.148282    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n" event={"ID":"a0687fd1-5202-43b7-af26-951a9fdf49c2","Type":"ContainerDied","Data":"af784010fa59cfd93f9f43c563572e7c9ad1cfdb24cfbabcd793140be704bce5"}
May 26 11:38:44.148306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.148303    2557 scope.go:117] "RemoveContainer" containerID="5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338"
May 26 11:38:44.148884 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.148868    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0687fd1-5202-43b7-af26-951a9fdf49c2-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:44.148973 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.148890    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxwhc\" (UniqueName: \"kubernetes.io/projected/a0687fd1-5202-43b7-af26-951a9fdf49c2-kube-api-access-vxwhc\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:38:44.149625 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.149596    2557 generic.go:358] "Generic (PLEG): container finished" podID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerID="2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e" exitCode=0
May 26 11:38:44.149716 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.149665    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerDied","Data":"2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e"}
May 26 11:38:44.151432 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.151414    2557 generic.go:358] "Generic (PLEG): container finished" podID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerID="5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20" exitCode=0
May 26 11:38:44.151541 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.151488    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"
May 26 11:38:44.151541 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.151499    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerDied","Data":"5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20"}
May 26 11:38:44.151541 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.151530    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6" event={"ID":"dc0cb65c-ebe3-4a03-a52a-677266309387","Type":"ContainerDied","Data":"0bdfaa4378fa9c96a8754872068377e687b49dd348edbdcdfd6490128f17e97f"}
May 26 11:38:44.152917 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.152897    2557 generic.go:358] "Generic (PLEG): container finished" podID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerID="eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619" exitCode=0
May 26 11:38:44.153003 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.152924    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerDied","Data":"eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619"}
May 26 11:38:44.158751 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.158720    2557 scope.go:117] "RemoveContainer" containerID="fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a"
May 26 11:38:44.166385 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.166370    2557 scope.go:117] "RemoveContainer" containerID="df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505"
May 26 11:38:44.175159 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.175140    2557 scope.go:117] "RemoveContainer" containerID="5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338"
May 26 11:38:44.175388 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:38:44.175367    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338\": container with ID starting with 5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338 not found: ID does not exist" containerID="5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338"
May 26 11:38:44.175470 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.175394    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338"} err="failed to get container status \"5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338\": rpc error: code = NotFound desc = could not find container \"5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338\": container with ID starting with 5c7ff96ca68d7766a2a80876a5be94c6771a69953f225cb526a1731854256338 not found: ID does not exist"
May 26 11:38:44.175470 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.175409    2557 scope.go:117] "RemoveContainer" containerID="fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a"
May 26 11:38:44.175669 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:38:44.175648    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a\": container with ID starting with fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a not found: ID does not exist" containerID="fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a"
May 26 11:38:44.175732 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.175674    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a"} err="failed to get container status \"fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a\": rpc error: code = NotFound desc = could not find container \"fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a\": container with ID starting with fcfef55cc7a06b1967491e689f98da418eda113d7b7a7ef197073dad3f81f88a not found: ID does not exist"
May 26 11:38:44.175732 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.175707    2557 scope.go:117] "RemoveContainer" containerID="df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505"
May 26 11:38:44.176386 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:38:44.175964    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505\": container with ID starting with df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505 not found: ID does not exist" containerID="df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505"
May 26 11:38:44.176535 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.176508    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505"} err="failed to get container status \"df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505\": rpc error: code = NotFound desc = could not find container \"df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505\": container with ID starting with df50c2f0ddd023fa0c12b5437bc86b07914c8fb1f40015a8c854f8b8297e2505 not found: ID does not exist"
May 26 11:38:44.176642 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.176628    2557 scope.go:117] "RemoveContainer" containerID="8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e"
May 26 11:38:44.195390 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.195374    2557 scope.go:117] "RemoveContainer" containerID="5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20"
May 26 11:38:44.205155 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.205140    2557 scope.go:117] "RemoveContainer" containerID="67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574"
May 26 11:38:44.211104 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.211059    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"]
May 26 11:38:44.212192 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.212176    2557 scope.go:117] "RemoveContainer" containerID="8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e"
May 26 11:38:44.212455 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:38:44.212436    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e\": container with ID starting with 8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e not found: ID does not exist" containerID="8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e"
May 26 11:38:44.212507 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.212469    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e"} err="failed to get container status \"8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e\": rpc error: code = NotFound desc = could not find container \"8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e\": container with ID starting with 8d89b802c1e8d2f613c86d6690d250642ecc0a97d399f08efc1ee03a2bb33c6e not found: ID does not exist"
May 26 11:38:44.212507 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.212487    2557 scope.go:117] "RemoveContainer" containerID="5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20"
May 26 11:38:44.212774 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:38:44.212757    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20\": container with ID starting with 5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20 not found: ID does not exist" containerID="5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20"
May 26 11:38:44.212813 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.212782    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20"} err="failed to get container status \"5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20\": rpc error: code = NotFound desc = could not find container \"5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20\": container with ID starting with 5260c03c46f9422987e5709b620aacc4de373ee15b1f198d0d1ec579710a1d20 not found: ID does not exist"
May 26 11:38:44.212813 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.212798    2557 scope.go:117] "RemoveContainer" containerID="67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574"
May 26 11:38:44.213020 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:38:44.213006    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574\": container with ID starting with 67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574 not found: ID does not exist" containerID="67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574"
May 26 11:38:44.213152 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.213022    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574"} err="failed to get container status \"67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574\": rpc error: code = NotFound desc = could not find container \"67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574\": container with ID starting with 67a09c45aa80e2461dbe554b3a23692aa40f5c4dd37804dc528ce521e8132574 not found: ID does not exist"
May 26 11:38:44.217229 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.217210    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ea074-predictor-75d9d67d7c-wwx7n"]
May 26 11:38:44.229364 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.229344    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"]
May 26 11:38:44.236497 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.236477    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ea074-predictor-6d59466565-vbtg6"]
May 26 11:38:44.798159 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.798130    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" path="/var/lib/kubelet/pods/a0687fd1-5202-43b7-af26-951a9fdf49c2/volumes"
May 26 11:38:44.798585 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:44.798573    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" path="/var/lib/kubelet/pods/dc0cb65c-ebe3-4a03-a52a-677266309387/volumes"
May 26 11:38:45.158801 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.158709    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerStarted","Data":"ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58"}
May 26 11:38:45.158801 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.158755    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerStarted","Data":"3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106"}
May 26 11:38:45.159305 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.159115    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:45.159305 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.159153    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:45.160771 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.160745    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:38:45.161633 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.161613    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerStarted","Data":"9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076"}
May 26 11:38:45.161742 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.161639    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerStarted","Data":"39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa"}
May 26 11:38:45.161864 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.161844    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:45.180818 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.180769    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podStartSLOduration=6.180758457 podStartE2EDuration="6.180758457s" podCreationTimestamp="2026-05-26 11:38:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:38:45.180452744 +0000 UTC m=+847.002836970" watchObservedRunningTime="2026-05-26 11:38:45.180758457 +0000 UTC m=+847.003142681"
May 26 11:38:45.210106 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:45.210046    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podStartSLOduration=6.210034561 podStartE2EDuration="6.210034561s" podCreationTimestamp="2026-05-26 11:38:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:38:45.207937403 +0000 UTC m=+847.030321619" watchObservedRunningTime="2026-05-26 11:38:45.210034561 +0000 UTC m=+847.032418817"
May 26 11:38:46.165705 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:46.165665    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:38:46.166119 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:46.165682    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:46.166906 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:46.166880    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:38:47.168414 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:47.168372    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:38:51.169995 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:51.169967    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:38:51.170542 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:51.170515    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:38:52.172345 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:52.172317    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:38:52.172892 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:38:52.172864    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:39:01.170982 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:01.170942    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:39:02.173695 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:02.173650    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:39:11.171088 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:11.171028    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:39:12.172900 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:12.172865    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:39:21.170620 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:21.170580    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:39:22.173298 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:22.173258    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:39:31.170653 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:31.170570    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:39:32.173135 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:32.173099    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:39:40.610993 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:40.610965    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:39:40.614894 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:40.614872    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:39:41.170881 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:41.170843    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:39:42.173224 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:42.173192    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:39:51.171639 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:39:51.171612    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:40:20.159342 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.159304    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"]
May 26 11:40:20.161594 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.159739    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" containerID="cri-o://39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa" gracePeriod=30
May 26 11:40:20.161594 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.159761    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kube-rbac-proxy" containerID="cri-o://9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076" gracePeriod=30
May 26 11:40:20.210473 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.210445    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"]
May 26 11:40:20.210778 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.210756    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" containerID="cri-o://3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106" gracePeriod=30
May 26 11:40:20.210851 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.210789    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kube-rbac-proxy" containerID="cri-o://ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58" gracePeriod=30
May 26 11:40:20.247513 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247478    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"]
May 26 11:40:20.247755 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247743    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kube-rbac-proxy"
May 26 11:40:20.247800 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247756    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kube-rbac-proxy"
May 26 11:40:20.247800 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247769    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="storage-initializer"
May 26 11:40:20.247800 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247774    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="storage-initializer"
May 26 11:40:20.247800 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247781    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kube-rbac-proxy"
May 26 11:40:20.247800 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247786    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kube-rbac-proxy"
May 26 11:40:20.247800 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247795    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container"
May 26 11:40:20.247800 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247800    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247808    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="storage-initializer"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247813    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="storage-initializer"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247824    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247829    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247867    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kube-rbac-proxy"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247875    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kube-rbac-proxy"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247881    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc0cb65c-ebe3-4a03-a52a-677266309387" containerName="kserve-container"
May 26 11:40:20.248021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.247888    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0687fd1-5202-43b7-af26-951a9fdf49c2" containerName="kserve-container"
May 26 11:40:20.250594 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.250579    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.253414 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.253389    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-a937a-predictor-serving-cert\""
May 26 11:40:20.253749 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.253733    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-a937a-kube-rbac-proxy-sar-config\""
May 26 11:40:20.262818 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.262795    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"]
May 26 11:40:20.323562 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.323534    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead048f4-f464-4e54-909c-46352edeb48c-message-dumper-raw-a937a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.323691 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.323604    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtmx\" (UniqueName: \"kubernetes.io/projected/ead048f4-f464-4e54-909c-46352edeb48c-kube-api-access-2xtmx\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.323740 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.323684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.418584 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.418504    2557 generic.go:358] "Generic (PLEG): container finished" podID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerID="9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076" exitCode=2
May 26 11:40:20.418584 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.418576    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerDied","Data":"9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076"}
May 26 11:40:20.420352 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.420332    2557 generic.go:358] "Generic (PLEG): container finished" podID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerID="ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58" exitCode=2
May 26 11:40:20.420447 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.420368    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerDied","Data":"ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58"}
May 26 11:40:20.424662 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.424644    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtmx\" (UniqueName: \"kubernetes.io/projected/ead048f4-f464-4e54-909c-46352edeb48c-kube-api-access-2xtmx\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.424755 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.424683    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.424811 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:20.424787    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-serving-cert: secret "message-dumper-raw-a937a-predictor-serving-cert" not found
May 26 11:40:20.424861 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.424809    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead048f4-f464-4e54-909c-46352edeb48c-message-dumper-raw-a937a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.424861 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:20.424848    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls podName:ead048f4-f464-4e54-909c-46352edeb48c nodeName:}" failed. No retries permitted until 2026-05-26 11:40:20.924828221 +0000 UTC m=+942.747212430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls") pod "message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" (UID: "ead048f4-f464-4e54-909c-46352edeb48c") : secret "message-dumper-raw-a937a-predictor-serving-cert" not found
May 26 11:40:20.425394 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.425376    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead048f4-f464-4e54-909c-46352edeb48c-message-dumper-raw-a937a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.433724 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.433699    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtmx\" (UniqueName: \"kubernetes.io/projected/ead048f4-f464-4e54-909c-46352edeb48c-kube-api-access-2xtmx\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.928430 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.928376    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:20.930700 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:20.930679    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls\") pod \"message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") " pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:21.160120 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:21.160065    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:21.166263 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:21.166219    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
May 26 11:40:21.170845 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:21.170813    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
May 26 11:40:21.281822 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:21.281790    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"]
May 26 11:40:21.283975 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:40:21.283938    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead048f4_f464_4e54_909c_46352edeb48c.slice/crio-d99c7504e695cfb9fb86c389e3f6faa6b193a9eda2731605faef3da4e6c3f9be WatchSource:0}: Error finding container d99c7504e695cfb9fb86c389e3f6faa6b193a9eda2731605faef3da4e6c3f9be: Status 404 returned error can't find the container with id d99c7504e695cfb9fb86c389e3f6faa6b193a9eda2731605faef3da4e6c3f9be
May 26 11:40:21.427296 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:21.427261    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" event={"ID":"ead048f4-f464-4e54-909c-46352edeb48c","Type":"ContainerStarted","Data":"d99c7504e695cfb9fb86c389e3f6faa6b193a9eda2731605faef3da4e6c3f9be"}
May 26 11:40:22.169182 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:22.169139    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 11:40:22.173481 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:22.173459    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 11:40:23.433947 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.433913    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" event={"ID":"ead048f4-f464-4e54-909c-46352edeb48c","Type":"ContainerStarted","Data":"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"}
May 26 11:40:23.433947 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.433946    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" event={"ID":"ead048f4-f464-4e54-909c-46352edeb48c","Type":"ContainerStarted","Data":"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"}
May 26 11:40:23.434358 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.434069    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:23.451757 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.451514    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" podStartSLOduration=2.019943069 podStartE2EDuration="3.451499436s" podCreationTimestamp="2026-05-26 11:40:20 +0000 UTC" firstStartedPulling="2026-05-26 11:40:21.285684416 +0000 UTC m=+943.108068622" lastFinishedPulling="2026-05-26 11:40:22.717240783 +0000 UTC m=+944.539624989" observedRunningTime="2026-05-26 11:40:23.450416208 +0000 UTC m=+945.272800435" watchObservedRunningTime="2026-05-26 11:40:23.451499436 +0000 UTC m=+945.273883651"
May 26 11:40:23.599697 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.599676    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:40:23.749162 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.749089    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44559c0-b91b-4c6a-be3a-c48d06807825-proxy-tls\") pod \"d44559c0-b91b-4c6a-be3a-c48d06807825\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") "
May 26 11:40:23.749162 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.749127    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfn4v\" (UniqueName: \"kubernetes.io/projected/d44559c0-b91b-4c6a-be3a-c48d06807825-kube-api-access-pfn4v\") pod \"d44559c0-b91b-4c6a-be3a-c48d06807825\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") "
May 26 11:40:23.749359 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.749174    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44559c0-b91b-4c6a-be3a-c48d06807825-isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"d44559c0-b91b-4c6a-be3a-c48d06807825\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") "
May 26 11:40:23.749359 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.749201    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44559c0-b91b-4c6a-be3a-c48d06807825-kserve-provision-location\") pod \"d44559c0-b91b-4c6a-be3a-c48d06807825\" (UID: \"d44559c0-b91b-4c6a-be3a-c48d06807825\") "
May 26 11:40:23.749565 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.749530    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44559c0-b91b-4c6a-be3a-c48d06807825-isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config") pod "d44559c0-b91b-4c6a-be3a-c48d06807825" (UID: "d44559c0-b91b-4c6a-be3a-c48d06807825"). InnerVolumeSpecName "isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:40:23.749634 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.749530    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44559c0-b91b-4c6a-be3a-c48d06807825-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d44559c0-b91b-4c6a-be3a-c48d06807825" (UID: "d44559c0-b91b-4c6a-be3a-c48d06807825"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:40:23.751159 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.751136    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44559c0-b91b-4c6a-be3a-c48d06807825-kube-api-access-pfn4v" (OuterVolumeSpecName: "kube-api-access-pfn4v") pod "d44559c0-b91b-4c6a-be3a-c48d06807825" (UID: "d44559c0-b91b-4c6a-be3a-c48d06807825"). InnerVolumeSpecName "kube-api-access-pfn4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:40:23.751242 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.751160    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44559c0-b91b-4c6a-be3a-c48d06807825-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d44559c0-b91b-4c6a-be3a-c48d06807825" (UID: "d44559c0-b91b-4c6a-be3a-c48d06807825"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:40:23.849677 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.849652    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44559c0-b91b-4c6a-be3a-c48d06807825-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:23.849677 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.849676    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfn4v\" (UniqueName: \"kubernetes.io/projected/d44559c0-b91b-4c6a-be3a-c48d06807825-kube-api-access-pfn4v\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:23.849819 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.849687    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44559c0-b91b-4c6a-be3a-c48d06807825-isvc-xgboost-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:23.849819 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:23.849699    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44559c0-b91b-4c6a-be3a-c48d06807825-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:24.351962 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.351941    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:40:24.438494 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.438427    2557 generic.go:358] "Generic (PLEG): container finished" podID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerID="39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa" exitCode=0
May 26 11:40:24.438494 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.438472    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerDied","Data":"39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa"}
May 26 11:40:24.438909 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.438510    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5" event={"ID":"d44559c0-b91b-4c6a-be3a-c48d06807825","Type":"ContainerDied","Data":"d4a78799f87824148cc0caddbfa9f06779ca552e2acff8b2982adea7d0923758"}
May 26 11:40:24.438909 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.438511    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"
May 26 11:40:24.438909 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.438524    2557 scope.go:117] "RemoveContainer" containerID="9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076"
May 26 11:40:24.440278 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.440252    2557 generic.go:358] "Generic (PLEG): container finished" podID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerID="3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106" exitCode=0
May 26 11:40:24.440396 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.440285    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerDied","Data":"3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106"}
May 26 11:40:24.440396 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.440327    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc" event={"ID":"f8002f55-cba5-4399-8c87-c8b7a27561b1","Type":"ContainerDied","Data":"4dbd3939d8f63f48cc431affa66f7e81e41a19bfc27553ada7f43fdaee6b455f"}
May 26 11:40:24.440396 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.440340    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"
May 26 11:40:24.440831 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.440812    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:24.442501 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.442482    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:24.449125 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.449104    2557 scope.go:117] "RemoveContainer" containerID="39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa"
May 26 11:40:24.453635 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.453616    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzn7j\" (UniqueName: \"kubernetes.io/projected/f8002f55-cba5-4399-8c87-c8b7a27561b1-kube-api-access-rzn7j\") pod \"f8002f55-cba5-4399-8c87-c8b7a27561b1\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") "
May 26 11:40:24.453711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.453652    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8002f55-cba5-4399-8c87-c8b7a27561b1-proxy-tls\") pod \"f8002f55-cba5-4399-8c87-c8b7a27561b1\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") "
May 26 11:40:24.453711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.453703    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8002f55-cba5-4399-8c87-c8b7a27561b1-kserve-provision-location\") pod \"f8002f55-cba5-4399-8c87-c8b7a27561b1\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") "
May 26 11:40:24.453807 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.453726    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8002f55-cba5-4399-8c87-c8b7a27561b1-isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") pod \"f8002f55-cba5-4399-8c87-c8b7a27561b1\" (UID: \"f8002f55-cba5-4399-8c87-c8b7a27561b1\") "
May 26 11:40:24.454031 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.454009    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8002f55-cba5-4399-8c87-c8b7a27561b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8002f55-cba5-4399-8c87-c8b7a27561b1" (UID: "f8002f55-cba5-4399-8c87-c8b7a27561b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:40:24.454145 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.454064    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8002f55-cba5-4399-8c87-c8b7a27561b1-isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config") pod "f8002f55-cba5-4399-8c87-c8b7a27561b1" (UID: "f8002f55-cba5-4399-8c87-c8b7a27561b1"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:40:24.455583 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.455559    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8002f55-cba5-4399-8c87-c8b7a27561b1-kube-api-access-rzn7j" (OuterVolumeSpecName: "kube-api-access-rzn7j") pod "f8002f55-cba5-4399-8c87-c8b7a27561b1" (UID: "f8002f55-cba5-4399-8c87-c8b7a27561b1"). InnerVolumeSpecName "kube-api-access-rzn7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:40:24.455675 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.455637    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8002f55-cba5-4399-8c87-c8b7a27561b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8002f55-cba5-4399-8c87-c8b7a27561b1" (UID: "f8002f55-cba5-4399-8c87-c8b7a27561b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:40:24.456251 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.456238    2557 scope.go:117] "RemoveContainer" containerID="2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e"
May 26 11:40:24.463588 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.463566    2557 scope.go:117] "RemoveContainer" containerID="9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076"
May 26 11:40:24.463805 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:24.463789    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076\": container with ID starting with 9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076 not found: ID does not exist" containerID="9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076"
May 26 11:40:24.463859 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.463812    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076"} err="failed to get container status \"9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076\": rpc error: code = NotFound desc = could not find container \"9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076\": container with ID starting with 9203d91e1a92e459e1cfd033ceec4d5647ad7df2720e5db2f0b00cb8a6039076 not found: ID does not exist"
May 26 11:40:24.463859 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.463828    2557 scope.go:117] "RemoveContainer" containerID="39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa"
May 26 11:40:24.464016 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:24.464002    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa\": container with ID starting with 39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa not found: ID does not exist" containerID="39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa"
May 26 11:40:24.464058 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.464019    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa"} err="failed to get container status \"39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa\": rpc error: code = NotFound desc = could not find container \"39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa\": container with ID starting with 39c5c90dd27a674fb4bec0c84f8795be956d729c57d88c89881c7b230e9480fa not found: ID does not exist"
May 26 11:40:24.464058 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.464033    2557 scope.go:117] "RemoveContainer" containerID="2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e"
May 26 11:40:24.464255 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:24.464236    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e\": container with ID starting with 2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e not found: ID does not exist" containerID="2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e"
May 26 11:40:24.464302 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.464257    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e"} err="failed to get container status \"2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e\": rpc error: code = NotFound desc = could not find container \"2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e\": container with ID starting with 2fc7e890673b7980bab8d3725393845dca297391cb67eabef30bc3a42db75e8e not found: ID does not exist"
May 26 11:40:24.464302 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.464269    2557 scope.go:117] "RemoveContainer" containerID="ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58"
May 26 11:40:24.470444 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.470428    2557 scope.go:117] "RemoveContainer" containerID="3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106"
May 26 11:40:24.476483 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.476461    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"]
May 26 11:40:24.479837 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.479820    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-019fa-predictor-855d9c6b9c-nh2d5"]
May 26 11:40:24.479887 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.479857    2557 scope.go:117] "RemoveContainer" containerID="eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619"
May 26 11:40:24.485926 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.485911    2557 scope.go:117] "RemoveContainer" containerID="ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58"
May 26 11:40:24.486148 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:24.486132    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58\": container with ID starting with ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58 not found: ID does not exist" containerID="ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58"
May 26 11:40:24.486198 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.486152    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58"} err="failed to get container status \"ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58\": rpc error: code = NotFound desc = could not find container \"ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58\": container with ID starting with ad69e35d31c0d3b04aa0a0a2dd5f95332aef6b8def7379585a6433e1dada0f58 not found: ID does not exist"
May 26 11:40:24.486198 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.486172    2557 scope.go:117] "RemoveContainer" containerID="3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106"
May 26 11:40:24.486387 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:24.486370    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106\": container with ID starting with 3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106 not found: ID does not exist" containerID="3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106"
May 26 11:40:24.486449 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.486396    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106"} err="failed to get container status \"3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106\": rpc error: code = NotFound desc = could not find container \"3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106\": container with ID starting with 3c55aad76d8584324431ab03838bc30d8693057708f55c66f5be5e523b277106 not found: ID does not exist"
May 26 11:40:24.486449 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.486419    2557 scope.go:117] "RemoveContainer" containerID="eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619"
May 26 11:40:24.486623 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:24.486609    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619\": container with ID starting with eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619 not found: ID does not exist" containerID="eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619"
May 26 11:40:24.486657 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.486626    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619"} err="failed to get container status \"eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619\": rpc error: code = NotFound desc = could not find container \"eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619\": container with ID starting with eb6ac873f08346aa05053baddbdf1c4ae07841646da3ae07d24bbde4461d1619 not found: ID does not exist"
May 26 11:40:24.555125 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.555102    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzn7j\" (UniqueName: \"kubernetes.io/projected/f8002f55-cba5-4399-8c87-c8b7a27561b1-kube-api-access-rzn7j\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:24.555125 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.555122    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8002f55-cba5-4399-8c87-c8b7a27561b1-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:24.555270 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.555132    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8002f55-cba5-4399-8c87-c8b7a27561b1-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:24.555270 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.555141    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8002f55-cba5-4399-8c87-c8b7a27561b1-isvc-sklearn-graph-raw-hpa-019fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:40:24.762466 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.762440    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"]
May 26 11:40:24.766311 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.766290    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-019fa-predictor-786c6c85c7-tpnqc"]
May 26 11:40:24.802457 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.802428    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" path="/var/lib/kubelet/pods/d44559c0-b91b-4c6a-be3a-c48d06807825/volumes"
May 26 11:40:24.802885 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:24.802872    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" path="/var/lib/kubelet/pods/f8002f55-cba5-4399-8c87-c8b7a27561b1/volumes"
May 26 11:40:31.451621 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:31.451594    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:40:40.251347 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251312    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"]
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251695    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251713    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251727    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251735    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251750    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kube-rbac-proxy"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251758    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kube-rbac-proxy"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251769    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="storage-initializer"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251776    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="storage-initializer"
May 26 11:40:40.251794 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251796    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kube-rbac-proxy"
May 26 11:40:40.252321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251805    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kube-rbac-proxy"
May 26 11:40:40.252321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251816    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="storage-initializer"
May 26 11:40:40.252321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251825    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="storage-initializer"
May 26 11:40:40.252321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251892    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kube-rbac-proxy"
May 26 11:40:40.252321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251904    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f8002f55-cba5-4399-8c87-c8b7a27561b1" containerName="kserve-container"
May 26 11:40:40.252321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251913    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kube-rbac-proxy"
May 26 11:40:40.252321 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.251925    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="d44559c0-b91b-4c6a-be3a-c48d06807825" containerName="kserve-container"
May 26 11:40:40.259573 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.259552    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.262231 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.262203    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\""
May 26 11:40:40.262345 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.262208    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-a937a-predictor-serving-cert\""
May 26 11:40:40.264288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.264265    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"]
May 26 11:40:40.268064 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:40.268034    2557 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa97e7de_0a77_4917_b327_35b62f812be4.slice\": RecentStats: unable to find data in memory cache]"
May 26 11:40:40.352511 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.352489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa97e7de-0a77-4917-b327-35b62f812be4-isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.352615 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.352519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa97e7de-0a77-4917-b327-35b62f812be4-kserve-provision-location\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.352615 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.352550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc889\" (UniqueName: \"kubernetes.io/projected/aa97e7de-0a77-4917-b327-35b62f812be4-kube-api-access-jc889\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.352615 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.352576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.453776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.453748    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jc889\" (UniqueName: \"kubernetes.io/projected/aa97e7de-0a77-4917-b327-35b62f812be4-kube-api-access-jc889\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.453915 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.453781    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.453915 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.453826    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa97e7de-0a77-4917-b327-35b62f812be4-isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.453915 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.453855    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa97e7de-0a77-4917-b327-35b62f812be4-kserve-provision-location\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.454107 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:40.453915    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-serving-cert: secret "isvc-logger-raw-a937a-predictor-serving-cert" not found
May 26 11:40:40.454107 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:40:40.453983    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls podName:aa97e7de-0a77-4917-b327-35b62f812be4 nodeName:}" failed. No retries permitted until 2026-05-26 11:40:40.953963812 +0000 UTC m=+962.776348028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls") pod "isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" (UID: "aa97e7de-0a77-4917-b327-35b62f812be4") : secret "isvc-logger-raw-a937a-predictor-serving-cert" not found
May 26 11:40:40.454286 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.454268    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa97e7de-0a77-4917-b327-35b62f812be4-kserve-provision-location\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.454572 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.454547    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa97e7de-0a77-4917-b327-35b62f812be4-isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.465332 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.465316    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc889\" (UniqueName: \"kubernetes.io/projected/aa97e7de-0a77-4917-b327-35b62f812be4-kube-api-access-jc889\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.957379 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.957344    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:40.959703 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:40.959675    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls\") pod \"isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:41.171137 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:41.171108    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:41.293411 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:41.293385    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"]
May 26 11:40:41.295604 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:40:41.295578    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa97e7de_0a77_4917_b327_35b62f812be4.slice/crio-136f47bf0f4a79bb921e39d5c80b6af13cb72a8f8708074c231d4d025f1ac650 WatchSource:0}: Error finding container 136f47bf0f4a79bb921e39d5c80b6af13cb72a8f8708074c231d4d025f1ac650: Status 404 returned error can't find the container with id 136f47bf0f4a79bb921e39d5c80b6af13cb72a8f8708074c231d4d025f1ac650
May 26 11:40:41.486741 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:41.486666    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerStarted","Data":"ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85"}
May 26 11:40:41.486741 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:41.486703    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerStarted","Data":"136f47bf0f4a79bb921e39d5c80b6af13cb72a8f8708074c231d4d025f1ac650"}
May 26 11:40:45.497692 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:45.497663    2557 generic.go:358] "Generic (PLEG): container finished" podID="aa97e7de-0a77-4917-b327-35b62f812be4" containerID="ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85" exitCode=0
May 26 11:40:45.498024 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:45.497736    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerDied","Data":"ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85"}
May 26 11:40:46.502320 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:46.502283    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerStarted","Data":"94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa"}
May 26 11:40:46.502669 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:46.502327    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerStarted","Data":"6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271"}
May 26 11:40:46.502669 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:46.502340    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerStarted","Data":"f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0"}
May 26 11:40:46.502669 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:46.502569    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:46.524234 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:46.524184    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podStartSLOduration=6.524169106 podStartE2EDuration="6.524169106s" podCreationTimestamp="2026-05-26 11:40:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:40:46.52234088 +0000 UTC m=+968.344725108" watchObservedRunningTime="2026-05-26 11:40:46.524169106 +0000 UTC m=+968.346553331"
May 26 11:40:47.505044 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:47.505004    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:47.505399 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:47.505053    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:47.506448 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:47.506412    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:40:47.507031 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:47.507005    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:40:48.507946 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:48.507896    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:40:48.508380 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:48.508351    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:40:53.511479 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:53.511450    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:40:53.512052 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:53.512023    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:40:53.512383 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:40:53.512358    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:41:03.512021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:03.511928    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:41:03.512500 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:03.512362    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:41:13.512562 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:13.512518    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:41:13.512996 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:13.512975    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:41:23.511963 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:23.511919    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:41:23.512423 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:23.512349    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:41:33.512594 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:33.512556    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:41:33.513070 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:33.513041    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:41:43.511994 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:43.511944    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:41:43.512531 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:43.512393    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:41:53.513220 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:53.513191    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:41:53.513589 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:41:53.513252    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:42:05.296497 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.296454    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4_ead048f4-f464-4e54-909c-46352edeb48c/kserve-container/0.log"
May 26 11:42:05.494529 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.494499    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"]
May 26 11:42:05.494870 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.494834    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" containerID="cri-o://f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0" gracePeriod=30
May 26 11:42:05.494958 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.494864    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" containerID="cri-o://94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa" gracePeriod=30
May 26 11:42:05.494958 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.494898    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy" containerID="cri-o://6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271" gracePeriod=30
May 26 11:42:05.536435 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.536404    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"]
May 26 11:42:05.539680 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.539661    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.542268 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.542248    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\""
May 26 11:42:05.542268 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.542258    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-15879-predictor-serving-cert\""
May 26 11:42:05.553576 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.553526    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"]
May 26 11:42:05.612729 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.612705    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"]
May 26 11:42:05.612991 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.612964    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kserve-container" containerID="cri-o://aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419" gracePeriod=30
May 26 11:42:05.613146 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.612980    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kube-rbac-proxy" containerID="cri-o://85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee" gracePeriod=30
May 26 11:42:05.642879 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.642855    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.642973 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.642891    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48js7\" (UniqueName: \"kubernetes.io/projected/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kube-api-access-48js7\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.642973 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.642923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.642973 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.642963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.712604 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.712574    2557 generic.go:358] "Generic (PLEG): container finished" podID="aa97e7de-0a77-4917-b327-35b62f812be4" containerID="6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271" exitCode=2
May 26 11:42:05.712702 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.712626    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerDied","Data":"6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271"}
May 26 11:42:05.744063 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.744038    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.744192 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.744089    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-48js7\" (UniqueName: \"kubernetes.io/projected/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kube-api-access-48js7\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.744192 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.744128    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.744192 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.744147    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.744360 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:05.744257    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-serving-cert: secret "isvc-sklearn-scale-raw-15879-predictor-serving-cert" not found
May 26 11:42:05.744360 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:05.744351    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls podName:b9da2639-8a5d-4e8d-9db3-0aa42612abfc nodeName:}" failed. No retries permitted until 2026-05-26 11:42:06.244330009 +0000 UTC m=+1048.066714231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls") pod "isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" (UID: "b9da2639-8a5d-4e8d-9db3-0aa42612abfc") : secret "isvc-sklearn-scale-raw-15879-predictor-serving-cert" not found
May 26 11:42:05.744569 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.744538    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.744803 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.744780    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.757645 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.757618    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48js7\" (UniqueName: \"kubernetes.io/projected/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kube-api-access-48js7\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:05.861932 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.861903    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:42:05.945745 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.945723    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls\") pod \"ead048f4-f464-4e54-909c-46352edeb48c\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") "
May 26 11:42:05.945850 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.945763    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xtmx\" (UniqueName: \"kubernetes.io/projected/ead048f4-f464-4e54-909c-46352edeb48c-kube-api-access-2xtmx\") pod \"ead048f4-f464-4e54-909c-46352edeb48c\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") "
May 26 11:42:05.945850 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.945792    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead048f4-f464-4e54-909c-46352edeb48c-message-dumper-raw-a937a-kube-rbac-proxy-sar-config\") pod \"ead048f4-f464-4e54-909c-46352edeb48c\" (UID: \"ead048f4-f464-4e54-909c-46352edeb48c\") "
May 26 11:42:05.946169 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.946146    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead048f4-f464-4e54-909c-46352edeb48c-message-dumper-raw-a937a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-raw-a937a-kube-rbac-proxy-sar-config") pod "ead048f4-f464-4e54-909c-46352edeb48c" (UID: "ead048f4-f464-4e54-909c-46352edeb48c"). InnerVolumeSpecName "message-dumper-raw-a937a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:42:05.947791 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.947774    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead048f4-f464-4e54-909c-46352edeb48c-kube-api-access-2xtmx" (OuterVolumeSpecName: "kube-api-access-2xtmx") pod "ead048f4-f464-4e54-909c-46352edeb48c" (UID: "ead048f4-f464-4e54-909c-46352edeb48c"). InnerVolumeSpecName "kube-api-access-2xtmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:42:05.947844 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:05.947802    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ead048f4-f464-4e54-909c-46352edeb48c" (UID: "ead048f4-f464-4e54-909c-46352edeb48c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:42:06.046270 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.046243    2557 reconciler_common.go:299] "Volume detached for volume \"message-dumper-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead048f4-f464-4e54-909c-46352edeb48c-message-dumper-raw-a937a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:42:06.046270 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.046266    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead048f4-f464-4e54-909c-46352edeb48c-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:42:06.046419 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.046277    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtmx\" (UniqueName: \"kubernetes.io/projected/ead048f4-f464-4e54-909c-46352edeb48c-kube-api-access-2xtmx\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:42:06.247157 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.247091    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:06.249287 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.249270    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls\") pod \"isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:06.449122 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.449088    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:06.573386 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.573347    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"]
May 26 11:42:06.575554 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:42:06.575521    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9da2639_8a5d_4e8d_9db3_0aa42612abfc.slice/crio-461f2435b45e3ffc399c1ee65e0568daf225205070b1b76d583879a6ca26a31b WatchSource:0}: Error finding container 461f2435b45e3ffc399c1ee65e0568daf225205070b1b76d583879a6ca26a31b: Status 404 returned error can't find the container with id 461f2435b45e3ffc399c1ee65e0568daf225205070b1b76d583879a6ca26a31b
May 26 11:42:06.717349 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.717313    2557 generic.go:358] "Generic (PLEG): container finished" podID="ead048f4-f464-4e54-909c-46352edeb48c" containerID="85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee" exitCode=2
May 26 11:42:06.717349 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.717342    2557 generic.go:358] "Generic (PLEG): container finished" podID="ead048f4-f464-4e54-909c-46352edeb48c" containerID="aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419" exitCode=2
May 26 11:42:06.717556 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.717395    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"
May 26 11:42:06.717556 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.717393    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" event={"ID":"ead048f4-f464-4e54-909c-46352edeb48c","Type":"ContainerDied","Data":"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"}
May 26 11:42:06.717556 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.717438    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" event={"ID":"ead048f4-f464-4e54-909c-46352edeb48c","Type":"ContainerDied","Data":"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"}
May 26 11:42:06.717556 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.717453    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4" event={"ID":"ead048f4-f464-4e54-909c-46352edeb48c","Type":"ContainerDied","Data":"d99c7504e695cfb9fb86c389e3f6faa6b193a9eda2731605faef3da4e6c3f9be"}
May 26 11:42:06.717556 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.717475    2557 scope.go:117] "RemoveContainer" containerID="85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"
May 26 11:42:06.718825 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.718799    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerStarted","Data":"c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9"}
May 26 11:42:06.718950 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.718834    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerStarted","Data":"461f2435b45e3ffc399c1ee65e0568daf225205070b1b76d583879a6ca26a31b"}
May 26 11:42:06.725503 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.725425    2557 scope.go:117] "RemoveContainer" containerID="aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"
May 26 11:42:06.732421 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.732403    2557 scope.go:117] "RemoveContainer" containerID="85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"
May 26 11:42:06.732670 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:06.732650    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee\": container with ID starting with 85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee not found: ID does not exist" containerID="85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"
May 26 11:42:06.732795 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.732674    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"} err="failed to get container status \"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee\": rpc error: code = NotFound desc = could not find container \"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee\": container with ID starting with 85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee not found: ID does not exist"
May 26 11:42:06.732795 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.732691    2557 scope.go:117] "RemoveContainer" containerID="aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"
May 26 11:42:06.732956 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:06.732939    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419\": container with ID starting with aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419 not found: ID does not exist" containerID="aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"
May 26 11:42:06.732998 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.732963    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"} err="failed to get container status \"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419\": rpc error: code = NotFound desc = could not find container \"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419\": container with ID starting with aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419 not found: ID does not exist"
May 26 11:42:06.732998 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.732981    2557 scope.go:117] "RemoveContainer" containerID="85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"
May 26 11:42:06.733224 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.733202    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee"} err="failed to get container status \"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee\": rpc error: code = NotFound desc = could not find container \"85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee\": container with ID starting with 85602c78cf4e828d208ce5f8bea3297af24d3448e7043bfe89c9939cfe8f14ee not found: ID does not exist"
May 26 11:42:06.733301 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.733227    2557 scope.go:117] "RemoveContainer" containerID="aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"
May 26 11:42:06.733475 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.733458    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419"} err="failed to get container status \"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419\": rpc error: code = NotFound desc = could not find container \"aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419\": container with ID starting with aa90999ce9af36dfdb8194ba381597c40e1817d861c348d6a8082534d3409419 not found: ID does not exist"
May 26 11:42:06.753888 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.753865    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"]
May 26 11:42:06.758795 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.758743    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a937a-predictor-75bcd59b59-2xqn4"]
May 26 11:42:06.802550 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:06.802522    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead048f4-f464-4e54-909c-46352edeb48c" path="/var/lib/kubelet/pods/ead048f4-f464-4e54-909c-46352edeb48c/volumes"
May 26 11:42:08.509049 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:08.509005    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 26 11:42:09.729169 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:09.729137    2557 generic.go:358] "Generic (PLEG): container finished" podID="aa97e7de-0a77-4917-b327-35b62f812be4" containerID="f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0" exitCode=0
May 26 11:42:09.729500 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:09.729192    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerDied","Data":"f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0"}
May 26 11:42:10.732886 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:10.732855    2557 generic.go:358] "Generic (PLEG): container finished" podID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerID="c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9" exitCode=0
May 26 11:42:10.733217 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:10.732907    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerDied","Data":"c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9"}
May 26 11:42:11.737748 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:11.737711    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerStarted","Data":"fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd"}
May 26 11:42:11.737748 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:11.737754    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerStarted","Data":"c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d"}
May 26 11:42:11.738179 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:11.737961    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:11.774940 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:11.774897    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podStartSLOduration=6.774883207 podStartE2EDuration="6.774883207s" podCreationTimestamp="2026-05-26 11:42:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:42:11.774104426 +0000 UTC m=+1053.596488652" watchObservedRunningTime="2026-05-26 11:42:11.774883207 +0000 UTC m=+1053.597267433"
May 26 11:42:12.741172 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:12.741138    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:12.742352 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:12.742324    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:42:13.508915 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:13.508878    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 26 11:42:13.512199 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:13.512164    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:42:13.512524 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:13.512505    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:42:13.743812 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:13.743773    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:42:18.508349 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:18.508305    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 26 11:42:18.508788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:18.508420    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:42:18.748012 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:18.747980    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:42:18.748506 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:18.748481    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:42:23.508588 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:23.508544    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 26 11:42:23.511953 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:23.511930    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:42:23.512296 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:23.512269    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:42:28.508508 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:28.508422    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 26 11:42:28.749166 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:28.749131    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:42:33.509027 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:33.508990    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 26 11:42:33.512278 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:33.512253    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
May 26 11:42:33.512391 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:33.512375    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:42:33.512715 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:33.512686    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 11:42:33.512822 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:33.512800    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:42:35.647903 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.647878    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:42:35.757145 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.757059    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc889\" (UniqueName: \"kubernetes.io/projected/aa97e7de-0a77-4917-b327-35b62f812be4-kube-api-access-jc889\") pod \"aa97e7de-0a77-4917-b327-35b62f812be4\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") "
May 26 11:42:35.757145 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.757131    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa97e7de-0a77-4917-b327-35b62f812be4-isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\") pod \"aa97e7de-0a77-4917-b327-35b62f812be4\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") "
May 26 11:42:35.757297 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.757184    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls\") pod \"aa97e7de-0a77-4917-b327-35b62f812be4\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") "
May 26 11:42:35.757297 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.757206    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa97e7de-0a77-4917-b327-35b62f812be4-kserve-provision-location\") pod \"aa97e7de-0a77-4917-b327-35b62f812be4\" (UID: \"aa97e7de-0a77-4917-b327-35b62f812be4\") "
May 26 11:42:35.757508 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.757477    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa97e7de-0a77-4917-b327-35b62f812be4-isvc-logger-raw-a937a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-a937a-kube-rbac-proxy-sar-config") pod "aa97e7de-0a77-4917-b327-35b62f812be4" (UID: "aa97e7de-0a77-4917-b327-35b62f812be4"). InnerVolumeSpecName "isvc-logger-raw-a937a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:42:35.757622 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.757513    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa97e7de-0a77-4917-b327-35b62f812be4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa97e7de-0a77-4917-b327-35b62f812be4" (UID: "aa97e7de-0a77-4917-b327-35b62f812be4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:35.759111 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.759089    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa97e7de-0a77-4917-b327-35b62f812be4" (UID: "aa97e7de-0a77-4917-b327-35b62f812be4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:42:35.759111 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.759093    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa97e7de-0a77-4917-b327-35b62f812be4-kube-api-access-jc889" (OuterVolumeSpecName: "kube-api-access-jc889") pod "aa97e7de-0a77-4917-b327-35b62f812be4" (UID: "aa97e7de-0a77-4917-b327-35b62f812be4"). InnerVolumeSpecName "kube-api-access-jc889". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:42:35.804398 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.804373    2557 generic.go:358] "Generic (PLEG): container finished" podID="aa97e7de-0a77-4917-b327-35b62f812be4" containerID="94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa" exitCode=0
May 26 11:42:35.804520 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.804455    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerDied","Data":"94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa"}
May 26 11:42:35.804520 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.804466    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"
May 26 11:42:35.804520 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.804490    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq" event={"ID":"aa97e7de-0a77-4917-b327-35b62f812be4","Type":"ContainerDied","Data":"136f47bf0f4a79bb921e39d5c80b6af13cb72a8f8708074c231d4d025f1ac650"}
May 26 11:42:35.804520 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.804505    2557 scope.go:117] "RemoveContainer" containerID="94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa"
May 26 11:42:35.811946 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.811735    2557 scope.go:117] "RemoveContainer" containerID="6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271"
May 26 11:42:35.818421 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.818403    2557 scope.go:117] "RemoveContainer" containerID="f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0"
May 26 11:42:35.825266 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.825251    2557 scope.go:117] "RemoveContainer" containerID="ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85"
May 26 11:42:35.831690 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.831674    2557 scope.go:117] "RemoveContainer" containerID="94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa"
May 26 11:42:35.831937 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:35.831902    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa\": container with ID starting with 94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa not found: ID does not exist" containerID="94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa"
May 26 11:42:35.832021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.831938    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa"} err="failed to get container status \"94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa\": rpc error: code = NotFound desc = could not find container \"94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa\": container with ID starting with 94a9e524846f3ed6a3977a08fb699fc80035bec5e359cc32a5c72684b60f6daa not found: ID does not exist"
May 26 11:42:35.832021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.831960    2557 scope.go:117] "RemoveContainer" containerID="6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271"
May 26 11:42:35.832155 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.832138    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"]
May 26 11:42:35.832205 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:35.832191    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271\": container with ID starting with 6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271 not found: ID does not exist" containerID="6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271"
May 26 11:42:35.832240 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.832212    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271"} err="failed to get container status \"6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271\": rpc error: code = NotFound desc = could not find container \"6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271\": container with ID starting with 6480bec5360d54506c1de9e46d99b5fc65bacca8d0aacdd6c20bd25819c45271 not found: ID does not exist"
May 26 11:42:35.832240 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.832227    2557 scope.go:117] "RemoveContainer" containerID="f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0"
May 26 11:42:35.832430 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:35.832416    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0\": container with ID starting with f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0 not found: ID does not exist" containerID="f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0"
May 26 11:42:35.832467 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.832434    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0"} err="failed to get container status \"f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0\": rpc error: code = NotFound desc = could not find container \"f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0\": container with ID starting with f721e678bc4d375c43823902153d78ed9f126c1d55671a6ebc99a8b8426e4fd0 not found: ID does not exist"
May 26 11:42:35.832467 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.832446    2557 scope.go:117] "RemoveContainer" containerID="ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85"
May 26 11:42:35.832628 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:42:35.832615    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85\": container with ID starting with ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85 not found: ID does not exist" containerID="ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85"
May 26 11:42:35.832664 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.832631    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85"} err="failed to get container status \"ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85\": rpc error: code = NotFound desc = could not find container \"ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85\": container with ID starting with ee529ae3f753c1fbf0849602d4b40ef99037f6fef0d3e1d3c4d71a633ccfce85 not found: ID does not exist"
May 26 11:42:35.840305 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.840286    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a937a-predictor-76c755b66c-l7cqq"]
May 26 11:42:35.857773 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.857755    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa97e7de-0a77-4917-b327-35b62f812be4-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:42:35.857773 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.857774    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa97e7de-0a77-4917-b327-35b62f812be4-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:42:35.857887 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.857784    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc889\" (UniqueName: \"kubernetes.io/projected/aa97e7de-0a77-4917-b327-35b62f812be4-kube-api-access-jc889\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:42:35.857887 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:35.857795    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa97e7de-0a77-4917-b327-35b62f812be4-isvc-logger-raw-a937a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:42:36.798597 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:36.798567    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" path="/var/lib/kubelet/pods/aa97e7de-0a77-4917-b327-35b62f812be4/volumes"
May 26 11:42:38.748938 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:38.748904    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:42:48.748907 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:48.748867    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:42:58.748875 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:42:58.748837    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:43:08.748835 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:43:08.748795    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:43:18.748973 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:43:18.748939    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:43:28.749327 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:43:28.749287    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:43:37.795524 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:43:37.795486    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:43:47.795461 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:43:47.795422    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:43:57.796495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:43:57.796451    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:44:07.796234 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:07.796196    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:44:17.796941 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:17.796913    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:44:25.783249 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.783217    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"]
May 26 11:44:25.783653 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.783565    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" containerID="cri-o://c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d" gracePeriod=30
May 26 11:44:25.783748 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.783614    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kube-rbac-proxy" containerID="cri-o://fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd" gracePeriod=30
May 26 11:44:25.985273 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985247    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"]
May 26 11:44:25.985500 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985487    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container"
May 26 11:44:25.985545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985501    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container"
May 26 11:44:25.985545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985510    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="storage-initializer"
May 26 11:44:25.985545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985516    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="storage-initializer"
May 26 11:44:25.985545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985523    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy"
May 26 11:44:25.985545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985529    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy"
May 26 11:44:25.985545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985536    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent"
May 26 11:44:25.985545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985541    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985551    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kserve-container"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985556    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kserve-container"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985568    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kube-rbac-proxy"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985573    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kube-rbac-proxy"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985613    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kserve-container"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985621    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kube-rbac-proxy"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985627    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="ead048f4-f464-4e54-909c-46352edeb48c" containerName="kube-rbac-proxy"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985635    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="agent"
May 26 11:44:25.985776 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.985641    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa97e7de-0a77-4917-b327-35b62f812be4" containerName="kserve-container"
May 26 11:44:25.988444 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.988429    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:25.992130 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.992107    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-287c88-predictor-serving-cert\""
May 26 11:44:25.992341 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:25.992326    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-287c88-kube-rbac-proxy-sar-config\""
May 26 11:44:26.003969 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.003947    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"]
May 26 11:44:26.096927 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.096859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2822bd4b-a3f3-4452-9575-28b8faf2931c-isvc-primary-287c88-kube-rbac-proxy-sar-config\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.096927 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.096895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2822bd4b-a3f3-4452-9575-28b8faf2931c-proxy-tls\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.096927 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.096913    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghw5h\" (UniqueName: \"kubernetes.io/projected/2822bd4b-a3f3-4452-9575-28b8faf2931c-kube-api-access-ghw5h\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.097141 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.096931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2822bd4b-a3f3-4452-9575-28b8faf2931c-kserve-provision-location\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.102561 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.102537    2557 generic.go:358] "Generic (PLEG): container finished" podID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerID="fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd" exitCode=2
May 26 11:44:26.102659 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.102575    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerDied","Data":"fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd"}
May 26 11:44:26.198140 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.198116    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2822bd4b-a3f3-4452-9575-28b8faf2931c-isvc-primary-287c88-kube-rbac-proxy-sar-config\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.198248 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.198147    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2822bd4b-a3f3-4452-9575-28b8faf2931c-proxy-tls\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.198248 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.198167    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ghw5h\" (UniqueName: \"kubernetes.io/projected/2822bd4b-a3f3-4452-9575-28b8faf2931c-kube-api-access-ghw5h\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.198248 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.198188    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2822bd4b-a3f3-4452-9575-28b8faf2931c-kserve-provision-location\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.198596 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.198574    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2822bd4b-a3f3-4452-9575-28b8faf2931c-kserve-provision-location\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.198755 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.198738    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2822bd4b-a3f3-4452-9575-28b8faf2931c-isvc-primary-287c88-kube-rbac-proxy-sar-config\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.200562 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.200541    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2822bd4b-a3f3-4452-9575-28b8faf2931c-proxy-tls\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.209486 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.209469    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghw5h\" (UniqueName: \"kubernetes.io/projected/2822bd4b-a3f3-4452-9575-28b8faf2931c-kube-api-access-ghw5h\") pod \"isvc-primary-287c88-predictor-dbc49469b-lssxz\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") " pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.298111 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.298070    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:26.420610 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.420583    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"]
May 26 11:44:26.422262 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:44:26.422232    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2822bd4b_a3f3_4452_9575_28b8faf2931c.slice/crio-1ad9176e8786b7bb8f21ea0d2ed58b8024aad75b19d345c6ec632b29f1f62ca1 WatchSource:0}: Error finding container 1ad9176e8786b7bb8f21ea0d2ed58b8024aad75b19d345c6ec632b29f1f62ca1: Status 404 returned error can't find the container with id 1ad9176e8786b7bb8f21ea0d2ed58b8024aad75b19d345c6ec632b29f1f62ca1
May 26 11:44:26.423944 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:26.423930    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:44:27.106497 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:27.106458    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerStarted","Data":"2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25"}
May 26 11:44:27.106497 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:27.106502    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerStarted","Data":"1ad9176e8786b7bb8f21ea0d2ed58b8024aad75b19d345c6ec632b29f1f62ca1"}
May 26 11:44:27.796471 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:27.796428    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 11:44:28.744171 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:28.744132    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 11:44:31.118891 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:31.118859    2557 generic.go:358] "Generic (PLEG): container finished" podID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerID="2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25" exitCode=0
May 26 11:44:31.119299 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:31.118896    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerDied","Data":"2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25"}
May 26 11:44:32.123149 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:32.123109    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerStarted","Data":"ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167"}
May 26 11:44:32.123149 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:32.123145    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerStarted","Data":"886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430"}
May 26 11:44:32.123560 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:32.123370    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:32.147777 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:32.147734    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podStartSLOduration=7.147722854 podStartE2EDuration="7.147722854s" podCreationTimestamp="2026-05-26 11:44:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:44:32.146237491 +0000 UTC m=+1193.968621718" watchObservedRunningTime="2026-05-26 11:44:32.147722854 +0000 UTC m=+1193.970107079"
May 26 11:44:33.126277 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:33.126242    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:33.127397 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:33.127365    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:44:33.745009 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:33.744969    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 11:44:34.122519 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.122496    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:44:34.129779 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.129754    2557 generic.go:358] "Generic (PLEG): container finished" podID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerID="c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d" exitCode=0
May 26 11:44:34.130068 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.129832    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerDied","Data":"c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d"}
May 26 11:44:34.130068 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.129869    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6" event={"ID":"b9da2639-8a5d-4e8d-9db3-0aa42612abfc","Type":"ContainerDied","Data":"461f2435b45e3ffc399c1ee65e0568daf225205070b1b76d583879a6ca26a31b"}
May 26 11:44:34.130068 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.129886    2557 scope.go:117] "RemoveContainer" containerID="fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd"
May 26 11:44:34.130068 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.129838    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"
May 26 11:44:34.130288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.130251    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:44:34.136689 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.136674    2557 scope.go:117] "RemoveContainer" containerID="c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d"
May 26 11:44:34.144268 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.144249    2557 scope.go:117] "RemoveContainer" containerID="c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9"
May 26 11:44:34.152713 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.152695    2557 scope.go:117] "RemoveContainer" containerID="fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd"
May 26 11:44:34.152982 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:44:34.152965    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd\": container with ID starting with fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd not found: ID does not exist" containerID="fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd"
May 26 11:44:34.153041 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.152987    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd"} err="failed to get container status \"fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd\": rpc error: code = NotFound desc = could not find container \"fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd\": container with ID starting with fef30c586db30d15a252f809eade89d8ff835bca39bcce8f796211bf29e355fd not found: ID does not exist"
May 26 11:44:34.153041 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.153002    2557 scope.go:117] "RemoveContainer" containerID="c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d"
May 26 11:44:34.153269 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:44:34.153249    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d\": container with ID starting with c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d not found: ID does not exist" containerID="c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d"
May 26 11:44:34.153359 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.153277    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d"} err="failed to get container status \"c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d\": rpc error: code = NotFound desc = could not find container \"c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d\": container with ID starting with c16cdb596c96f4423d11cd3334b221a7cf370990605b533f632bf712e5f5588d not found: ID does not exist"
May 26 11:44:34.153359 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.153294    2557 scope.go:117] "RemoveContainer" containerID="c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9"
May 26 11:44:34.153538 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:44:34.153519    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9\": container with ID starting with c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9 not found: ID does not exist" containerID="c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9"
May 26 11:44:34.153584 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.153543    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9"} err="failed to get container status \"c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9\": rpc error: code = NotFound desc = could not find container \"c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9\": container with ID starting with c249a3581da0b982e7aa4d957549a4bb54409cce37ad85b27c56eecbbe6c87e9 not found: ID does not exist"
May 26 11:44:34.253179 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.253120    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kserve-provision-location\") pod \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") "
May 26 11:44:34.253179 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.253153    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls\") pod \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") "
May 26 11:44:34.253320 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.253189    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48js7\" (UniqueName: \"kubernetes.io/projected/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kube-api-access-48js7\") pod \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") "
May 26 11:44:34.253320 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.253254    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\") pod \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\" (UID: \"b9da2639-8a5d-4e8d-9db3-0aa42612abfc\") "
May 26 11:44:34.253411 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.253398    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9da2639-8a5d-4e8d-9db3-0aa42612abfc" (UID: "b9da2639-8a5d-4e8d-9db3-0aa42612abfc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:44:34.253490 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.253472    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:44:34.253724 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.253689    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config") pod "b9da2639-8a5d-4e8d-9db3-0aa42612abfc" (UID: "b9da2639-8a5d-4e8d-9db3-0aa42612abfc"). InnerVolumeSpecName "isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:44:34.255169 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.255151    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kube-api-access-48js7" (OuterVolumeSpecName: "kube-api-access-48js7") pod "b9da2639-8a5d-4e8d-9db3-0aa42612abfc" (UID: "b9da2639-8a5d-4e8d-9db3-0aa42612abfc"). InnerVolumeSpecName "kube-api-access-48js7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:44:34.255599 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.255583    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9da2639-8a5d-4e8d-9db3-0aa42612abfc" (UID: "b9da2639-8a5d-4e8d-9db3-0aa42612abfc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:44:34.354567 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.354543    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48js7\" (UniqueName: \"kubernetes.io/projected/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-kube-api-access-48js7\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:44:34.354567 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.354564    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-isvc-sklearn-scale-raw-15879-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:44:34.354689 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.354575    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9da2639-8a5d-4e8d-9db3-0aa42612abfc-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:44:34.453668 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.453647    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"]
May 26 11:44:34.457264 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.457235    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-15879-predictor-6fcdc78f79-stpl6"]
May 26 11:44:34.798506 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:34.798474    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" path="/var/lib/kubelet/pods/b9da2639-8a5d-4e8d-9db3-0aa42612abfc/volumes"
May 26 11:44:39.134969 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:39.134943    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:44:39.135564 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:39.135535    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:44:40.630438 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:40.630411    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:44:40.634106 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:40.634070    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:44:49.136426 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:49.136393    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:44:59.135743 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:44:59.135703    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:45:09.136456 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:09.136419    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:45:19.135570 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:19.135535    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:45:29.136106 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:29.136053    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 11:45:39.136214 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:39.136184    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:45:45.977318 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977288    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"]
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977534    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container"
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977546    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container"
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977556    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kube-rbac-proxy"
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977561    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kube-rbac-proxy"
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977568    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="storage-initializer"
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977573    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="storage-initializer"
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977657    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kserve-container"
May 26 11:45:45.977830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.977669    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b9da2639-8a5d-4e8d-9db3-0aa42612abfc" containerName="kube-rbac-proxy"
May 26 11:45:45.980642 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.980626    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:45.983306 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.983287    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-287c88-kube-rbac-proxy-sar-config\""
May 26 11:45:45.983429 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.983312    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-287c88\""
May 26 11:45:45.983584 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.983570    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-287c88-predictor-serving-cert\""
May 26 11:45:45.983667 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.983579    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 11:45:45.984516 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.984500    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-287c88-dockercfg-7klwp\""
May 26 11:45:45.988414 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:45.988392    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"]
May 26 11:45:46.032326 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.032301    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70bc9027-29b0-4a3d-9024-15bea0b0442b-kserve-provision-location\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.032431 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.032337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bc9027-29b0-4a3d-9024-15bea0b0442b-proxy-tls\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.032431 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.032367    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-cabundle-cert\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.032431 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.032419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nt8d\" (UniqueName: \"kubernetes.io/projected/70bc9027-29b0-4a3d-9024-15bea0b0442b-kube-api-access-6nt8d\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.032539 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.032453    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-isvc-secondary-287c88-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.132935 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.132912    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6nt8d\" (UniqueName: \"kubernetes.io/projected/70bc9027-29b0-4a3d-9024-15bea0b0442b-kube-api-access-6nt8d\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.133032 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.132942    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-isvc-secondary-287c88-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.133163 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.133142    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70bc9027-29b0-4a3d-9024-15bea0b0442b-kserve-provision-location\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.133225 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.133189    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bc9027-29b0-4a3d-9024-15bea0b0442b-proxy-tls\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.133291 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.133221    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-cabundle-cert\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.133409 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.133393    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70bc9027-29b0-4a3d-9024-15bea0b0442b-kserve-provision-location\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.133545 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.133530    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-isvc-secondary-287c88-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.133775 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.133760    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-cabundle-cert\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.135436 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.135421    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bc9027-29b0-4a3d-9024-15bea0b0442b-proxy-tls\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.141432 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.141414    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nt8d\" (UniqueName: \"kubernetes.io/projected/70bc9027-29b0-4a3d-9024-15bea0b0442b-kube-api-access-6nt8d\") pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") " pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.291689 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.291666    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:45:46.412472 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:46.412448    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"]
May 26 11:45:46.414849 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:45:46.414811    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bc9027_29b0_4a3d_9024_15bea0b0442b.slice/crio-8106b60a72d11ebeb08390f0d7315f1d151135423034fca93354a6abc40a10ee WatchSource:0}: Error finding container 8106b60a72d11ebeb08390f0d7315f1d151135423034fca93354a6abc40a10ee: Status 404 returned error can't find the container with id 8106b60a72d11ebeb08390f0d7315f1d151135423034fca93354a6abc40a10ee
May 26 11:45:47.332415 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:47.332376    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" event={"ID":"70bc9027-29b0-4a3d-9024-15bea0b0442b","Type":"ContainerStarted","Data":"8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d"}
May 26 11:45:47.332415 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:47.332415    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" event={"ID":"70bc9027-29b0-4a3d-9024-15bea0b0442b","Type":"ContainerStarted","Data":"8106b60a72d11ebeb08390f0d7315f1d151135423034fca93354a6abc40a10ee"}
May 26 11:45:48.336998 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:48.336976    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_70bc9027-29b0-4a3d-9024-15bea0b0442b/storage-initializer/0.log"
May 26 11:45:48.337280 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:48.337014    2557 generic.go:358] "Generic (PLEG): container finished" podID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerID="8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d" exitCode=1
May 26 11:45:48.337280 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:48.337096    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" event={"ID":"70bc9027-29b0-4a3d-9024-15bea0b0442b","Type":"ContainerDied","Data":"8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d"}
May 26 11:45:49.341253 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:49.341226    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_70bc9027-29b0-4a3d-9024-15bea0b0442b/storage-initializer/0.log"
May 26 11:45:49.341618 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:49.341297    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" event={"ID":"70bc9027-29b0-4a3d-9024-15bea0b0442b","Type":"ContainerStarted","Data":"0b47e352a028097ca7773cbb299109dd52e7eaaa29321d442fc848e8de4b946f"}
May 26 11:45:50.345237 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:50.345215    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_70bc9027-29b0-4a3d-9024-15bea0b0442b/storage-initializer/1.log"
May 26 11:45:50.345581 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:50.345564    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_70bc9027-29b0-4a3d-9024-15bea0b0442b/storage-initializer/0.log"
May 26 11:45:50.345623 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:50.345600    2557 generic.go:358] "Generic (PLEG): container finished" podID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerID="0b47e352a028097ca7773cbb299109dd52e7eaaa29321d442fc848e8de4b946f" exitCode=1
May 26 11:45:50.345688 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:50.345676    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" event={"ID":"70bc9027-29b0-4a3d-9024-15bea0b0442b","Type":"ContainerDied","Data":"0b47e352a028097ca7773cbb299109dd52e7eaaa29321d442fc848e8de4b946f"}
May 26 11:45:50.345723 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:50.345709    2557 scope.go:117] "RemoveContainer" containerID="8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d"
May 26 11:45:50.345961 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:50.345938    2557 scope.go:117] "RemoveContainer" containerID="8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d"
May 26 11:45:50.355655 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:45:50.355629    2557 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_kserve-ci-e2e-test_70bc9027-29b0-4a3d-9024-15bea0b0442b_0 in pod sandbox 8106b60a72d11ebeb08390f0d7315f1d151135423034fca93354a6abc40a10ee from index: no such id: '8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d'" containerID="8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d"
May 26 11:45:50.355727 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:45:50.355673    2557 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_kserve-ci-e2e-test_70bc9027-29b0-4a3d-9024-15bea0b0442b_0 in pod sandbox 8106b60a72d11ebeb08390f0d7315f1d151135423034fca93354a6abc40a10ee from index: no such id: '8a224be78fa64a6241b860d0291448871aab14f5ee3e1c88ec43f42b4c465b8d'; Skipping pod \"isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_kserve-ci-e2e-test(70bc9027-29b0-4a3d-9024-15bea0b0442b)\"" logger="UnhandledError"
May 26 11:45:50.356978 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:45:50.356960    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_kserve-ci-e2e-test(70bc9027-29b0-4a3d-9024-15bea0b0442b)\"" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b"
May 26 11:45:51.350115 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:45:51.350091    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_70bc9027-29b0-4a3d-9024-15bea0b0442b/storage-initializer/1.log"
May 26 11:45:51.350586 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:45:51.350565    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_kserve-ci-e2e-test(70bc9027-29b0-4a3d-9024-15bea0b0442b)\"" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b"
May 26 11:46:00.025295 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.025263    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"]
May 26 11:46:00.069885 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.069853    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"]
May 26 11:46:00.070264 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.070204    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container" containerID="cri-o://886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430" gracePeriod=30
May 26 11:46:00.070416 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.070258    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kube-rbac-proxy" containerID="cri-o://ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167" gracePeriod=30
May 26 11:46:00.154626 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.154597    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"]
May 26 11:46:00.159309 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.159292    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.161889 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.161871    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-a468f5\""
May 26 11:46:00.162032 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.161996    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-a468f5-dockercfg-bflp4\""
May 26 11:46:00.162107 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.161999    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a468f5-predictor-serving-cert\""
May 26 11:46:00.162153 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.162136    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\""
May 26 11:46:00.169182 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.169162    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"]
May 26 11:46:00.188743 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.188728    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_70bc9027-29b0-4a3d-9024-15bea0b0442b/storage-initializer/1.log"
May 26 11:46:00.188824 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.188785    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:46:00.231711 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231685    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-isvc-secondary-287c88-kube-rbac-proxy-sar-config\") pod \"70bc9027-29b0-4a3d-9024-15bea0b0442b\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") "
May 26 11:46:00.231802 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231741    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-cabundle-cert\") pod \"70bc9027-29b0-4a3d-9024-15bea0b0442b\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") "
May 26 11:46:00.231802 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231769    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bc9027-29b0-4a3d-9024-15bea0b0442b-proxy-tls\") pod \"70bc9027-29b0-4a3d-9024-15bea0b0442b\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") "
May 26 11:46:00.231802 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231786    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nt8d\" (UniqueName: \"kubernetes.io/projected/70bc9027-29b0-4a3d-9024-15bea0b0442b-kube-api-access-6nt8d\") pod \"70bc9027-29b0-4a3d-9024-15bea0b0442b\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") "
May 26 11:46:00.231951 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231828    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70bc9027-29b0-4a3d-9024-15bea0b0442b-kserve-provision-location\") pod \"70bc9027-29b0-4a3d-9024-15bea0b0442b\" (UID: \"70bc9027-29b0-4a3d-9024-15bea0b0442b\") "
May 26 11:46:00.231951 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.231951 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54a3da0-1040-424d-8e05-8b316008c49f-kserve-provision-location\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.232126 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.231974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5pb\" (UniqueName: \"kubernetes.io/projected/b54a3da0-1040-424d-8e05-8b316008c49f-kube-api-access-pt5pb\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.232126 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-cabundle-cert\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.232126 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.232261 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232068    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-isvc-secondary-287c88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-287c88-kube-rbac-proxy-sar-config") pod "70bc9027-29b0-4a3d-9024-15bea0b0442b" (UID: "70bc9027-29b0-4a3d-9024-15bea0b0442b"). InnerVolumeSpecName "isvc-secondary-287c88-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:46:00.232261 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232153    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bc9027-29b0-4a3d-9024-15bea0b0442b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70bc9027-29b0-4a3d-9024-15bea0b0442b" (UID: "70bc9027-29b0-4a3d-9024-15bea0b0442b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:00.232261 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232178    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70bc9027-29b0-4a3d-9024-15bea0b0442b" (UID: "70bc9027-29b0-4a3d-9024-15bea0b0442b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:46:00.232261 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232248    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-cabundle-cert\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:00.232390 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232262    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70bc9027-29b0-4a3d-9024-15bea0b0442b-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:00.232390 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.232280    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70bc9027-29b0-4a3d-9024-15bea0b0442b-isvc-secondary-287c88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:00.233720 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.233698    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bc9027-29b0-4a3d-9024-15bea0b0442b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70bc9027-29b0-4a3d-9024-15bea0b0442b" (UID: "70bc9027-29b0-4a3d-9024-15bea0b0442b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:46:00.233841 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.233821    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bc9027-29b0-4a3d-9024-15bea0b0442b-kube-api-access-6nt8d" (OuterVolumeSpecName: "kube-api-access-6nt8d") pod "70bc9027-29b0-4a3d-9024-15bea0b0442b" (UID: "70bc9027-29b0-4a3d-9024-15bea0b0442b"). InnerVolumeSpecName "kube-api-access-6nt8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:46:00.333332 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333309    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-cabundle-cert\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.333424 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333339    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.333424 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333392    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.333424 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333408    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54a3da0-1040-424d-8e05-8b316008c49f-kserve-provision-location\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.333574 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333426    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5pb\" (UniqueName: \"kubernetes.io/projected/b54a3da0-1040-424d-8e05-8b316008c49f-kube-api-access-pt5pb\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.333574 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333453    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70bc9027-29b0-4a3d-9024-15bea0b0442b-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:00.333574 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333462    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nt8d\" (UniqueName: \"kubernetes.io/projected/70bc9027-29b0-4a3d-9024-15bea0b0442b-kube-api-access-6nt8d\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:00.333574 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:00.333563    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-serving-cert: secret "isvc-init-fail-a468f5-predictor-serving-cert" not found
May 26 11:46:00.333775 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:00.333627    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls podName:b54a3da0-1040-424d-8e05-8b316008c49f nodeName:}" failed. No retries permitted until 2026-05-26 11:46:00.833607678 +0000 UTC m=+1282.655991884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls") pod "isvc-init-fail-a468f5-predictor-5689d84987-j62j2" (UID: "b54a3da0-1040-424d-8e05-8b316008c49f") : secret "isvc-init-fail-a468f5-predictor-serving-cert" not found
May 26 11:46:00.333775 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333715    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54a3da0-1040-424d-8e05-8b316008c49f-kserve-provision-location\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.333926 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.333906    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-cabundle-cert\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.334065 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.334044    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.344424 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.344404    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5pb\" (UniqueName: \"kubernetes.io/projected/b54a3da0-1040-424d-8e05-8b316008c49f-kube-api-access-pt5pb\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.378168 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.378145    2557 generic.go:358] "Generic (PLEG): container finished" podID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerID="ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167" exitCode=2
May 26 11:46:00.378269 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.378225    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerDied","Data":"ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167"}
May 26 11:46:00.379167 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.379152    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-287c88-predictor-f85b56fbc-fg5zv_70bc9027-29b0-4a3d-9024-15bea0b0442b/storage-initializer/1.log"
May 26 11:46:00.379281 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.379220    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv" event={"ID":"70bc9027-29b0-4a3d-9024-15bea0b0442b","Type":"ContainerDied","Data":"8106b60a72d11ebeb08390f0d7315f1d151135423034fca93354a6abc40a10ee"}
May 26 11:46:00.379281 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.379249    2557 scope.go:117] "RemoveContainer" containerID="0b47e352a028097ca7773cbb299109dd52e7eaaa29321d442fc848e8de4b946f"
May 26 11:46:00.379281 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.379272    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"
May 26 11:46:00.415028 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.415007    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"]
May 26 11:46:00.418681 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.418658    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-287c88-predictor-f85b56fbc-fg5zv"]
May 26 11:46:00.798224 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.798196    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b" path="/var/lib/kubelet/pods/70bc9027-29b0-4a3d-9024-15bea0b0442b/volumes"
May 26 11:46:00.837993 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.837965    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:00.840354 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:00.840334    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls\") pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") " pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:01.069539 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:01.069456    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:01.191746 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:01.191712    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"]
May 26 11:46:01.194767 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:46:01.194734    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54a3da0_1040_424d_8e05_8b316008c49f.slice/crio-4f2078315ce4fbe9632d8225a482c7c4f3923dd6ab7bb28acff0d48925e2de4a WatchSource:0}: Error finding container 4f2078315ce4fbe9632d8225a482c7c4f3923dd6ab7bb28acff0d48925e2de4a: Status 404 returned error can't find the container with id 4f2078315ce4fbe9632d8225a482c7c4f3923dd6ab7bb28acff0d48925e2de4a
May 26 11:46:01.383548 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:01.383443    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2" event={"ID":"b54a3da0-1040-424d-8e05-8b316008c49f","Type":"ContainerStarted","Data":"beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9"}
May 26 11:46:01.383548 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:01.383503    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2" event={"ID":"b54a3da0-1040-424d-8e05-8b316008c49f","Type":"ContainerStarted","Data":"4f2078315ce4fbe9632d8225a482c7c4f3923dd6ab7bb28acff0d48925e2de4a"}
May 26 11:46:04.001408 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.001388    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:46:04.063308 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.063279    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghw5h\" (UniqueName: \"kubernetes.io/projected/2822bd4b-a3f3-4452-9575-28b8faf2931c-kube-api-access-ghw5h\") pod \"2822bd4b-a3f3-4452-9575-28b8faf2931c\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") "
May 26 11:46:04.063438 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.063347    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2822bd4b-a3f3-4452-9575-28b8faf2931c-kserve-provision-location\") pod \"2822bd4b-a3f3-4452-9575-28b8faf2931c\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") "
May 26 11:46:04.063438 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.063371    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2822bd4b-a3f3-4452-9575-28b8faf2931c-proxy-tls\") pod \"2822bd4b-a3f3-4452-9575-28b8faf2931c\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") "
May 26 11:46:04.063438 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.063405    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2822bd4b-a3f3-4452-9575-28b8faf2931c-isvc-primary-287c88-kube-rbac-proxy-sar-config\") pod \"2822bd4b-a3f3-4452-9575-28b8faf2931c\" (UID: \"2822bd4b-a3f3-4452-9575-28b8faf2931c\") "
May 26 11:46:04.063698 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.063667    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2822bd4b-a3f3-4452-9575-28b8faf2931c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2822bd4b-a3f3-4452-9575-28b8faf2931c" (UID: "2822bd4b-a3f3-4452-9575-28b8faf2931c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:04.063803 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.063733    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2822bd4b-a3f3-4452-9575-28b8faf2931c-isvc-primary-287c88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-287c88-kube-rbac-proxy-sar-config") pod "2822bd4b-a3f3-4452-9575-28b8faf2931c" (UID: "2822bd4b-a3f3-4452-9575-28b8faf2931c"). InnerVolumeSpecName "isvc-primary-287c88-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:46:04.065386 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.065365    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2822bd4b-a3f3-4452-9575-28b8faf2931c-kube-api-access-ghw5h" (OuterVolumeSpecName: "kube-api-access-ghw5h") pod "2822bd4b-a3f3-4452-9575-28b8faf2931c" (UID: "2822bd4b-a3f3-4452-9575-28b8faf2931c"). InnerVolumeSpecName "kube-api-access-ghw5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:46:04.065480 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.065409    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2822bd4b-a3f3-4452-9575-28b8faf2931c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2822bd4b-a3f3-4452-9575-28b8faf2931c" (UID: "2822bd4b-a3f3-4452-9575-28b8faf2931c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:46:04.163904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.163851    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-287c88-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2822bd4b-a3f3-4452-9575-28b8faf2931c-isvc-primary-287c88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:04.163904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.163875    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghw5h\" (UniqueName: \"kubernetes.io/projected/2822bd4b-a3f3-4452-9575-28b8faf2931c-kube-api-access-ghw5h\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:04.163904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.163884    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2822bd4b-a3f3-4452-9575-28b8faf2931c-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:04.163904 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.163893    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2822bd4b-a3f3-4452-9575-28b8faf2931c-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:04.399164 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.399134    2557 generic.go:358] "Generic (PLEG): container finished" podID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerID="886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430" exitCode=0
May 26 11:46:04.399263 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.399176    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerDied","Data":"886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430"}
May 26 11:46:04.399263 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.399209    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"
May 26 11:46:04.399263 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.399219    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz" event={"ID":"2822bd4b-a3f3-4452-9575-28b8faf2931c","Type":"ContainerDied","Data":"1ad9176e8786b7bb8f21ea0d2ed58b8024aad75b19d345c6ec632b29f1f62ca1"}
May 26 11:46:04.399263 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.399236    2557 scope.go:117] "RemoveContainer" containerID="ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167"
May 26 11:46:04.408616 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.408598    2557 scope.go:117] "RemoveContainer" containerID="886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430"
May 26 11:46:04.415362 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.415345    2557 scope.go:117] "RemoveContainer" containerID="2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25"
May 26 11:46:04.422882 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.422862    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"]
May 26 11:46:04.423484 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.423434    2557 scope.go:117] "RemoveContainer" containerID="ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167"
May 26 11:46:04.423736 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:04.423717    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167\": container with ID starting with ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167 not found: ID does not exist" containerID="ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167"
May 26 11:46:04.423832 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.423743    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167"} err="failed to get container status \"ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167\": rpc error: code = NotFound desc = could not find container \"ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167\": container with ID starting with ea270a1f62f44a20c321e51ff551cd02a6bf2cab3bd7b1f7d9a5d7ba6b75f167 not found: ID does not exist"
May 26 11:46:04.423832 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.423762    2557 scope.go:117] "RemoveContainer" containerID="886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430"
May 26 11:46:04.424052 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:04.424030    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430\": container with ID starting with 886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430 not found: ID does not exist" containerID="886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430"
May 26 11:46:04.424131 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.424059    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430"} err="failed to get container status \"886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430\": rpc error: code = NotFound desc = could not find container \"886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430\": container with ID starting with 886ca293dfd28ef1ffd5a01ee61d6dafb6323aa15f2a4e16af766bdf5faa5430 not found: ID does not exist"
May 26 11:46:04.424131 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.424095    2557 scope.go:117] "RemoveContainer" containerID="2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25"
May 26 11:46:04.424580 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:04.424458    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25\": container with ID starting with 2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25 not found: ID does not exist" containerID="2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25"
May 26 11:46:04.424580 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.424490    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25"} err="failed to get container status \"2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25\": rpc error: code = NotFound desc = could not find container \"2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25\": container with ID starting with 2bbaa5817a1be0fc810afaa9f8652c5c31c3b17c15920706de9f22144c8fda25 not found: ID does not exist"
May 26 11:46:04.427028 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.427003    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-287c88-predictor-dbc49469b-lssxz"]
May 26 11:46:04.798829 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:04.798803    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" path="/var/lib/kubelet/pods/2822bd4b-a3f3-4452-9575-28b8faf2931c/volumes"
May 26 11:46:05.402999 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:05.402975    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_b54a3da0-1040-424d-8e05-8b316008c49f/storage-initializer/0.log"
May 26 11:46:05.403376 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:05.403013    2557 generic.go:358] "Generic (PLEG): container finished" podID="b54a3da0-1040-424d-8e05-8b316008c49f" containerID="beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9" exitCode=1
May 26 11:46:05.403376 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:05.403098    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2" event={"ID":"b54a3da0-1040-424d-8e05-8b316008c49f","Type":"ContainerDied","Data":"beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9"}
May 26 11:46:06.408365 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:06.408340    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_b54a3da0-1040-424d-8e05-8b316008c49f/storage-initializer/0.log"
May 26 11:46:06.408716 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:06.408423    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2" event={"ID":"b54a3da0-1040-424d-8e05-8b316008c49f","Type":"ContainerStarted","Data":"1509eeda57bcb946b3ea95ae943bbcfed743cb0661a2862bd476ecd5159db316"}
May 26 11:46:07.412828 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:07.412806    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_b54a3da0-1040-424d-8e05-8b316008c49f/storage-initializer/1.log"
May 26 11:46:07.413242 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:07.413228    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_b54a3da0-1040-424d-8e05-8b316008c49f/storage-initializer/0.log"
May 26 11:46:07.413297 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:07.413259    2557 generic.go:358] "Generic (PLEG): container finished" podID="b54a3da0-1040-424d-8e05-8b316008c49f" containerID="1509eeda57bcb946b3ea95ae943bbcfed743cb0661a2862bd476ecd5159db316" exitCode=1
May 26 11:46:07.413339 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:07.413317    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2" event={"ID":"b54a3da0-1040-424d-8e05-8b316008c49f","Type":"ContainerDied","Data":"1509eeda57bcb946b3ea95ae943bbcfed743cb0661a2862bd476ecd5159db316"}
May 26 11:46:07.413373 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:07.413358    2557 scope.go:117] "RemoveContainer" containerID="beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9"
May 26 11:46:07.413739 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:07.413722    2557 scope.go:117] "RemoveContainer" containerID="beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9"
May 26 11:46:07.423112 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:07.423067    2557 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_kserve-ci-e2e-test_b54a3da0-1040-424d-8e05-8b316008c49f_0 in pod sandbox 4f2078315ce4fbe9632d8225a482c7c4f3923dd6ab7bb28acff0d48925e2de4a from index: no such id: 'beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9'" containerID="beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9"
May 26 11:46:07.423182 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:07.423130    2557 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_kserve-ci-e2e-test_b54a3da0-1040-424d-8e05-8b316008c49f_0 in pod sandbox 4f2078315ce4fbe9632d8225a482c7c4f3923dd6ab7bb28acff0d48925e2de4a from index: no such id: 'beeb19349ade53c333045c3c239d9205a7c3026122f3605e416c85cc528583c9'; Skipping pod \"isvc-init-fail-a468f5-predictor-5689d84987-j62j2_kserve-ci-e2e-test(b54a3da0-1040-424d-8e05-8b316008c49f)\"" logger="UnhandledError"
May 26 11:46:07.424432 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:46:07.424412    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-a468f5-predictor-5689d84987-j62j2_kserve-ci-e2e-test(b54a3da0-1040-424d-8e05-8b316008c49f)\"" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2" podUID="b54a3da0-1040-424d-8e05-8b316008c49f"
May 26 11:46:08.417544 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:08.417516    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_b54a3da0-1040-424d-8e05-8b316008c49f/storage-initializer/1.log"
May 26 11:46:10.197115 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.197066    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"]
May 26 11:46:10.321947 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.321924    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_b54a3da0-1040-424d-8e05-8b316008c49f/storage-initializer/1.log"
May 26 11:46:10.322045 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.321992    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:10.354378 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354325    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"]
May 26 11:46:10.354601 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354588    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerName="storage-initializer"
May 26 11:46:10.354637 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354603    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerName="storage-initializer"
May 26 11:46:10.354637 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354614    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container"
May 26 11:46:10.354637 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354621    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container"
May 26 11:46:10.354637 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354627    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54a3da0-1040-424d-8e05-8b316008c49f" containerName="storage-initializer"
May 26 11:46:10.354637 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354634    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54a3da0-1040-424d-8e05-8b316008c49f" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354639    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54a3da0-1040-424d-8e05-8b316008c49f" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354644    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54a3da0-1040-424d-8e05-8b316008c49f" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354653    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kube-rbac-proxy"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354658    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kube-rbac-proxy"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354670    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354675    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354711    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b54a3da0-1040-424d-8e05-8b316008c49f" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354717    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354724    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kserve-container"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354730    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b54a3da0-1040-424d-8e05-8b316008c49f" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354736    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="2822bd4b-a3f3-4452-9575-28b8faf2931c" containerName="kube-rbac-proxy"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354772    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerName="storage-initializer"
May 26 11:46:10.354788 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354777    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerName="storage-initializer"
May 26 11:46:10.355184 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.354820    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="70bc9027-29b0-4a3d-9024-15bea0b0442b" containerName="storage-initializer"
May 26 11:46:10.358771 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.358753    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.362011 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.361992    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\""
May 26 11:46:10.362565 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.362546    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-4e4c6-predictor-serving-cert\""
May 26 11:46:10.363406 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.363393    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wtv9z\""
May 26 11:46:10.373448 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.373429    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"]
May 26 11:46:10.406185 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406166    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls\") pod \"b54a3da0-1040-424d-8e05-8b316008c49f\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") "
May 26 11:46:10.406287 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406217    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\") pod \"b54a3da0-1040-424d-8e05-8b316008c49f\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") "
May 26 11:46:10.406287 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406248    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-cabundle-cert\") pod \"b54a3da0-1040-424d-8e05-8b316008c49f\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") "
May 26 11:46:10.406403 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406299    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54a3da0-1040-424d-8e05-8b316008c49f-kserve-provision-location\") pod \"b54a3da0-1040-424d-8e05-8b316008c49f\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") "
May 26 11:46:10.406403 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406326    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt5pb\" (UniqueName: \"kubernetes.io/projected/b54a3da0-1040-424d-8e05-8b316008c49f-kube-api-access-pt5pb\") pod \"b54a3da0-1040-424d-8e05-8b316008c49f\" (UID: \"b54a3da0-1040-424d-8e05-8b316008c49f\") "
May 26 11:46:10.406498 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb67ebf-1d83-4854-8077-8ed885de3d89-kserve-provision-location\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.406549 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406513    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twv4b\" (UniqueName: \"kubernetes.io/projected/4eb67ebf-1d83-4854-8077-8ed885de3d89-kube-api-access-twv4b\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.406596 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406546    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eb67ebf-1d83-4854-8077-8ed885de3d89-proxy-tls\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.406596 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406562    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b54a3da0-1040-424d-8e05-8b316008c49f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b54a3da0-1040-424d-8e05-8b316008c49f" (UID: "b54a3da0-1040-424d-8e05-8b316008c49f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:10.406681 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406617    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-isvc-init-fail-a468f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-a468f5-kube-rbac-proxy-sar-config") pod "b54a3da0-1040-424d-8e05-8b316008c49f" (UID: "b54a3da0-1040-424d-8e05-8b316008c49f"). InnerVolumeSpecName "isvc-init-fail-a468f5-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:46:10.406681 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eb67ebf-1d83-4854-8077-8ed885de3d89-raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.406759 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406688    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b54a3da0-1040-424d-8e05-8b316008c49f" (UID: "b54a3da0-1040-424d-8e05-8b316008c49f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:46:10.406759 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406729    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-isvc-init-fail-a468f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:10.406759 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.406743    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b54a3da0-1040-424d-8e05-8b316008c49f-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:10.408294 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.408278    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b54a3da0-1040-424d-8e05-8b316008c49f" (UID: "b54a3da0-1040-424d-8e05-8b316008c49f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:46:10.408477 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.408458    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54a3da0-1040-424d-8e05-8b316008c49f-kube-api-access-pt5pb" (OuterVolumeSpecName: "kube-api-access-pt5pb") pod "b54a3da0-1040-424d-8e05-8b316008c49f" (UID: "b54a3da0-1040-424d-8e05-8b316008c49f"). InnerVolumeSpecName "kube-api-access-pt5pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:46:10.424122 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.424102    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a468f5-predictor-5689d84987-j62j2_b54a3da0-1040-424d-8e05-8b316008c49f/storage-initializer/1.log"
May 26 11:46:10.424192 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.424175    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2" event={"ID":"b54a3da0-1040-424d-8e05-8b316008c49f","Type":"ContainerDied","Data":"4f2078315ce4fbe9632d8225a482c7c4f3923dd6ab7bb28acff0d48925e2de4a"}
May 26 11:46:10.424226 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.424202    2557 scope.go:117] "RemoveContainer" containerID="1509eeda57bcb946b3ea95ae943bbcfed743cb0661a2862bd476ecd5159db316"
May 26 11:46:10.424226 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.424208    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"
May 26 11:46:10.461529 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.461473    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"]
May 26 11:46:10.467099 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.467059    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a468f5-predictor-5689d84987-j62j2"]
May 26 11:46:10.507298 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507276    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb67ebf-1d83-4854-8077-8ed885de3d89-kserve-provision-location\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.507387 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507317    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-twv4b\" (UniqueName: \"kubernetes.io/projected/4eb67ebf-1d83-4854-8077-8ed885de3d89-kube-api-access-twv4b\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.507387 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507336    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eb67ebf-1d83-4854-8077-8ed885de3d89-proxy-tls\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.507387 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507364    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eb67ebf-1d83-4854-8077-8ed885de3d89-raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.507564 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507400    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5pb\" (UniqueName: \"kubernetes.io/projected/b54a3da0-1040-424d-8e05-8b316008c49f-kube-api-access-pt5pb\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:10.507564 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507410    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b54a3da0-1040-424d-8e05-8b316008c49f-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:10.507564 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507419    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b54a3da0-1040-424d-8e05-8b316008c49f-cabundle-cert\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:46:10.507760 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507743    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb67ebf-1d83-4854-8077-8ed885de3d89-kserve-provision-location\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.508006 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.507990    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eb67ebf-1d83-4854-8077-8ed885de3d89-raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.509490 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.509471    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eb67ebf-1d83-4854-8077-8ed885de3d89-proxy-tls\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.516171 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.516153    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twv4b\" (UniqueName: \"kubernetes.io/projected/4eb67ebf-1d83-4854-8077-8ed885de3d89-kube-api-access-twv4b\") pod \"raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") " pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.668616 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.668596    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:10.788596 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.788567    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"]
May 26 11:46:10.792000 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:46:10.791975    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb67ebf_1d83_4854_8077_8ed885de3d89.slice/crio-d0b1168225762258b169c3c967aa219fff8dedeceff4d64d52d7ba8f9fefe9a0 WatchSource:0}: Error finding container d0b1168225762258b169c3c967aa219fff8dedeceff4d64d52d7ba8f9fefe9a0: Status 404 returned error can't find the container with id d0b1168225762258b169c3c967aa219fff8dedeceff4d64d52d7ba8f9fefe9a0
May 26 11:46:10.802760 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:10.799857    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b54a3da0-1040-424d-8e05-8b316008c49f" path="/var/lib/kubelet/pods/b54a3da0-1040-424d-8e05-8b316008c49f/volumes"
May 26 11:46:11.428441 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:11.428407    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerStarted","Data":"6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868"}
May 26 11:46:11.428846 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:11.428445    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerStarted","Data":"d0b1168225762258b169c3c967aa219fff8dedeceff4d64d52d7ba8f9fefe9a0"}
May 26 11:46:15.441282 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:15.441249    2557 generic.go:358] "Generic (PLEG): container finished" podID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerID="6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868" exitCode=0
May 26 11:46:15.441614 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:15.441291    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerDied","Data":"6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868"}
May 26 11:46:16.446119 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:16.446064    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerStarted","Data":"b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6"}
May 26 11:46:16.446119 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:16.446118    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerStarted","Data":"d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779"}
May 26 11:46:16.446599 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:16.446398    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:16.446599 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:16.446513    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:16.447585 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:16.447562    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:46:16.467238 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:16.467195    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podStartSLOduration=6.467184126 podStartE2EDuration="6.467184126s" podCreationTimestamp="2026-05-26 11:46:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:46:16.46601884 +0000 UTC m=+1298.288403064" watchObservedRunningTime="2026-05-26 11:46:16.467184126 +0000 UTC m=+1298.289568351"
May 26 11:46:17.449369 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:17.449333    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:46:22.453137 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:22.453109    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:46:22.453650 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:22.453622    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:46:32.453611 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:32.453564    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:46:42.453733 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:42.453695    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:46:52.453700 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:46:52.453663    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:47:02.453623 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:02.453538    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:47:12.453542 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:12.453502    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:47:22.454240 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:22.454214    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:47:30.449342 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.449307    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"]
May 26 11:47:30.449808 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.449633    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" containerID="cri-o://d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779" gracePeriod=30
May 26 11:47:30.449808 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.449649    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kube-rbac-proxy" containerID="cri-o://b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6" gracePeriod=30
May 26 11:47:30.534416 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.534390    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"]
May 26 11:47:30.537700 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.537684    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.540320 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.540299    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-51723-predictor-serving-cert\""
May 26 11:47:30.540412 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.540343    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\""
May 26 11:47:30.550549 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.550531    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"]
May 26 11:47:30.643730 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.643706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c1c88b-4d98-42c3-86b5-2895bbb980fc-proxy-tls\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.643839 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.643750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpdt\" (UniqueName: \"kubernetes.io/projected/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kube-api-access-vlpdt\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.643839 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.643769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kserve-provision-location\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.643839 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.643794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c1c88b-4d98-42c3-86b5-2895bbb980fc-raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.650950 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.650929    2557 generic.go:358] "Generic (PLEG): container finished" podID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerID="b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6" exitCode=2
May 26 11:47:30.651095 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.650995    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerDied","Data":"b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6"}
May 26 11:47:30.744961 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.744900    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpdt\" (UniqueName: \"kubernetes.io/projected/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kube-api-access-vlpdt\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.744961 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.744935    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kserve-provision-location\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.744961 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.744959    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c1c88b-4d98-42c3-86b5-2895bbb980fc-raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.745203 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.745002    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c1c88b-4d98-42c3-86b5-2895bbb980fc-proxy-tls\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.745373 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.745350    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kserve-provision-location\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.745610 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.745594    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c1c88b-4d98-42c3-86b5-2895bbb980fc-raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.747290 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.747270    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c1c88b-4d98-42c3-86b5-2895bbb980fc-proxy-tls\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.754012 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.753989    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpdt\" (UniqueName: \"kubernetes.io/projected/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kube-api-access-vlpdt\") pod \"raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.846950 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.846927    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:30.972267 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:30.972242    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"]
May 26 11:47:30.974555 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:47:30.974528    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c1c88b_4d98_42c3_86b5_2895bbb980fc.slice/crio-43888b199623d091944eac8220f82ae5e58ce577b12c0e39d15e3fab36d0a0eb WatchSource:0}: Error finding container 43888b199623d091944eac8220f82ae5e58ce577b12c0e39d15e3fab36d0a0eb: Status 404 returned error can't find the container with id 43888b199623d091944eac8220f82ae5e58ce577b12c0e39d15e3fab36d0a0eb
May 26 11:47:31.655446 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:31.655410    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerStarted","Data":"73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6"}
May 26 11:47:31.655810 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:31.655453    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerStarted","Data":"43888b199623d091944eac8220f82ae5e58ce577b12c0e39d15e3fab36d0a0eb"}
May 26 11:47:32.449844 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:32.449809    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
May 26 11:47:32.454530 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:32.454502    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 11:47:34.293452 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.293430    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:47:34.371977 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.371913    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eb67ebf-1d83-4854-8077-8ed885de3d89-proxy-tls\") pod \"4eb67ebf-1d83-4854-8077-8ed885de3d89\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") "
May 26 11:47:34.372115 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.372013    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eb67ebf-1d83-4854-8077-8ed885de3d89-raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\") pod \"4eb67ebf-1d83-4854-8077-8ed885de3d89\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") "
May 26 11:47:34.372115 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.372068    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb67ebf-1d83-4854-8077-8ed885de3d89-kserve-provision-location\") pod \"4eb67ebf-1d83-4854-8077-8ed885de3d89\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") "
May 26 11:47:34.372245 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.372144    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twv4b\" (UniqueName: \"kubernetes.io/projected/4eb67ebf-1d83-4854-8077-8ed885de3d89-kube-api-access-twv4b\") pod \"4eb67ebf-1d83-4854-8077-8ed885de3d89\" (UID: \"4eb67ebf-1d83-4854-8077-8ed885de3d89\") "
May 26 11:47:34.372356 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.372333    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb67ebf-1d83-4854-8077-8ed885de3d89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4eb67ebf-1d83-4854-8077-8ed885de3d89" (UID: "4eb67ebf-1d83-4854-8077-8ed885de3d89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:47:34.372395 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.372370    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb67ebf-1d83-4854-8077-8ed885de3d89-raw-sklearn-4e4c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-4e4c6-kube-rbac-proxy-sar-config") pod "4eb67ebf-1d83-4854-8077-8ed885de3d89" (UID: "4eb67ebf-1d83-4854-8077-8ed885de3d89"). InnerVolumeSpecName "raw-sklearn-4e4c6-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:47:34.373994 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.373971    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb67ebf-1d83-4854-8077-8ed885de3d89-kube-api-access-twv4b" (OuterVolumeSpecName: "kube-api-access-twv4b") pod "4eb67ebf-1d83-4854-8077-8ed885de3d89" (UID: "4eb67ebf-1d83-4854-8077-8ed885de3d89"). InnerVolumeSpecName "kube-api-access-twv4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:47:34.374097 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.374064    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb67ebf-1d83-4854-8077-8ed885de3d89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4eb67ebf-1d83-4854-8077-8ed885de3d89" (UID: "4eb67ebf-1d83-4854-8077-8ed885de3d89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:47:34.473537 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.473512    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb67ebf-1d83-4854-8077-8ed885de3d89-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:47:34.473537 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.473536    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twv4b\" (UniqueName: \"kubernetes.io/projected/4eb67ebf-1d83-4854-8077-8ed885de3d89-kube-api-access-twv4b\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:47:34.473666 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.473551    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eb67ebf-1d83-4854-8077-8ed885de3d89-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:47:34.473666 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.473565    2557 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eb67ebf-1d83-4854-8077-8ed885de3d89-raw-sklearn-4e4c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:47:34.664563 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.664495    2557 generic.go:358] "Generic (PLEG): container finished" podID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerID="d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779" exitCode=0
May 26 11:47:34.664563 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.664546    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerDied","Data":"d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779"}
May 26 11:47:34.664723 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.664572    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4" event={"ID":"4eb67ebf-1d83-4854-8077-8ed885de3d89","Type":"ContainerDied","Data":"d0b1168225762258b169c3c967aa219fff8dedeceff4d64d52d7ba8f9fefe9a0"}
May 26 11:47:34.664723 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.664575    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"
May 26 11:47:34.664723 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.664586    2557 scope.go:117] "RemoveContainer" containerID="b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6"
May 26 11:47:34.673029 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.673014    2557 scope.go:117] "RemoveContainer" containerID="d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779"
May 26 11:47:34.679595 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.679578    2557 scope.go:117] "RemoveContainer" containerID="6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868"
May 26 11:47:34.686792 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.686773    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"]
May 26 11:47:34.686855 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.686816    2557 scope.go:117] "RemoveContainer" containerID="b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6"
May 26 11:47:34.687091 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:47:34.687057    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6\": container with ID starting with b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6 not found: ID does not exist" containerID="b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6"
May 26 11:47:34.687176 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.687098    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6"} err="failed to get container status \"b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6\": rpc error: code = NotFound desc = could not find container \"b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6\": container with ID starting with b02aa4436a2207364df644d01aea10d69075cb10c98019bc19b65e84344505c6 not found: ID does not exist"
May 26 11:47:34.687176 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.687116    2557 scope.go:117] "RemoveContainer" containerID="d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779"
May 26 11:47:34.687404 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:47:34.687381    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779\": container with ID starting with d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779 not found: ID does not exist" containerID="d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779"
May 26 11:47:34.687451 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.687413    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779"} err="failed to get container status \"d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779\": rpc error: code = NotFound desc = could not find container \"d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779\": container with ID starting with d2056a824fbd53b7dffece48e403853ef16cd5046277b158590765378e132779 not found: ID does not exist"
May 26 11:47:34.687451 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.687435    2557 scope.go:117] "RemoveContainer" containerID="6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868"
May 26 11:47:34.687724 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:47:34.687708    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868\": container with ID starting with 6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868 not found: ID does not exist" containerID="6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868"
May 26 11:47:34.687766 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.687730    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868"} err="failed to get container status \"6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868\": rpc error: code = NotFound desc = could not find container \"6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868\": container with ID starting with 6b804a25ee494f75305e0fac621bf7664fee77b10bc3d1e3054c898a5df1d868 not found: ID does not exist"
May 26 11:47:34.693116 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.693098    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-4e4c6-predictor-6cb9cfb575-qsmj4"]
May 26 11:47:34.798413 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:34.798389    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" path="/var/lib/kubelet/pods/4eb67ebf-1d83-4854-8077-8ed885de3d89/volumes"
May 26 11:47:35.669396 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:35.669366    2557 generic.go:358] "Generic (PLEG): container finished" podID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerID="73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6" exitCode=0
May 26 11:47:35.669810 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:35.669443    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerDied","Data":"73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6"}
May 26 11:47:36.675180 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:36.675147    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerStarted","Data":"b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06"}
May 26 11:47:36.675180 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:36.675183    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerStarted","Data":"9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc"}
May 26 11:47:36.675631 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:36.675460    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:36.675631 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:36.675573    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:36.676830 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:36.676803    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:47:36.699531 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:36.699493    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podStartSLOduration=6.699481561 podStartE2EDuration="6.699481561s" podCreationTimestamp="2026-05-26 11:47:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:47:36.697068201 +0000 UTC m=+1378.519452426" watchObservedRunningTime="2026-05-26 11:47:36.699481561 +0000 UTC m=+1378.521865796"
May 26 11:47:37.678258 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:37.678214    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:47:42.682441 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:42.682411    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:47:42.682988 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:42.682963    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:47:52.683803 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:47:52.683765    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:48:02.682982 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:02.682941    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:48:12.682945 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:12.682908    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:48:22.683527 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:22.683487    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:48:32.683301 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:32.683263    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:48:42.683994 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:42.683962    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:48:50.619495 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:50.619463    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"]
May 26 11:48:50.620029 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:50.619837    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" containerID="cri-o://9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc" gracePeriod=30
May 26 11:48:50.620029 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:50.619884    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kube-rbac-proxy" containerID="cri-o://b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06" gracePeriod=30
May 26 11:48:50.866685 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:50.866651    2557 generic.go:358] "Generic (PLEG): container finished" podID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerID="b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06" exitCode=2
May 26 11:48:50.866844 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:50.866724    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerDied","Data":"b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06"}
May 26 11:48:52.679415 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:52.679375    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
May 26 11:48:52.683656 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:52.683634    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
May 26 11:48:54.458424 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.458402    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:48:54.606691 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.606653    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlpdt\" (UniqueName: \"kubernetes.io/projected/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kube-api-access-vlpdt\") pod \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") "
May 26 11:48:54.606835 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.606709    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kserve-provision-location\") pod \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") "
May 26 11:48:54.606835 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.606763    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c1c88b-4d98-42c3-86b5-2895bbb980fc-proxy-tls\") pod \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") "
May 26 11:48:54.606835 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.606790    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c1c88b-4d98-42c3-86b5-2895bbb980fc-raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\") pod \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\" (UID: \"35c1c88b-4d98-42c3-86b5-2895bbb980fc\") "
May 26 11:48:54.606991 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.606948    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35c1c88b-4d98-42c3-86b5-2895bbb980fc" (UID: "35c1c88b-4d98-42c3-86b5-2895bbb980fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:48:54.607211 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.607183    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c1c88b-4d98-42c3-86b5-2895bbb980fc-raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config") pod "35c1c88b-4d98-42c3-86b5-2895bbb980fc" (UID: "35c1c88b-4d98-42c3-86b5-2895bbb980fc"). InnerVolumeSpecName "raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:48:54.608698 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.608679    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kube-api-access-vlpdt" (OuterVolumeSpecName: "kube-api-access-vlpdt") pod "35c1c88b-4d98-42c3-86b5-2895bbb980fc" (UID: "35c1c88b-4d98-42c3-86b5-2895bbb980fc"). InnerVolumeSpecName "kube-api-access-vlpdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:48:54.608772 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.608724    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c1c88b-4d98-42c3-86b5-2895bbb980fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35c1c88b-4d98-42c3-86b5-2895bbb980fc" (UID: "35c1c88b-4d98-42c3-86b5-2895bbb980fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:48:54.707337 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.707310    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpdt\" (UniqueName: \"kubernetes.io/projected/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kube-api-access-vlpdt\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:48:54.707337 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.707334    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c1c88b-4d98-42c3-86b5-2895bbb980fc-kserve-provision-location\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:48:54.707493 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.707348    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c1c88b-4d98-42c3-86b5-2895bbb980fc-proxy-tls\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:48:54.707493 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.707364    2557 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c1c88b-4d98-42c3-86b5-2895bbb980fc-raw-sklearn-runtime-51723-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-203.ec2.internal\" DevicePath \"\""
May 26 11:48:54.877969 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.877908    2557 generic.go:358] "Generic (PLEG): container finished" podID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerID="9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc" exitCode=0
May 26 11:48:54.878097 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.877986    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"
May 26 11:48:54.878097 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.877996    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerDied","Data":"9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc"}
May 26 11:48:54.878097 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.878038    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj" event={"ID":"35c1c88b-4d98-42c3-86b5-2895bbb980fc","Type":"ContainerDied","Data":"43888b199623d091944eac8220f82ae5e58ce577b12c0e39d15e3fab36d0a0eb"}
May 26 11:48:54.878097 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.878056    2557 scope.go:117] "RemoveContainer" containerID="b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06"
May 26 11:48:54.885240 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.885210    2557 scope.go:117] "RemoveContainer" containerID="9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc"
May 26 11:48:54.891571 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.891556    2557 scope.go:117] "RemoveContainer" containerID="73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6"
May 26 11:48:54.898232 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.898202    2557 scope.go:117] "RemoveContainer" containerID="b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06"
May 26 11:48:54.898476 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:48:54.898456    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06\": container with ID starting with b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06 not found: ID does not exist" containerID="b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06"
May 26 11:48:54.898563 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.898483    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06"} err="failed to get container status \"b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06\": rpc error: code = NotFound desc = could not find container \"b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06\": container with ID starting with b1417fe1b8c1ff921ca791e2789da4dfbaef01931ad5c28316fa8b611840fd06 not found: ID does not exist"
May 26 11:48:54.898563 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.898499    2557 scope.go:117] "RemoveContainer" containerID="9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc"
May 26 11:48:54.898738 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:48:54.898721    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc\": container with ID starting with 9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc not found: ID does not exist" containerID="9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc"
May 26 11:48:54.898815 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.898743    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc"} err="failed to get container status \"9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc\": rpc error: code = NotFound desc = could not find container \"9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc\": container with ID starting with 9dcb9f75690601c84b4fd241cdfd73d63ccae959c62c897c07eaece99a858ccc not found: ID does not exist"
May 26 11:48:54.898815 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.898759    2557 scope.go:117] "RemoveContainer" containerID="73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6"
May 26 11:48:54.898815 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.898777    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"]
May 26 11:48:54.898978 ip-10-0-132-203 kubenswrapper[2557]: E0526 11:48:54.898958    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6\": container with ID starting with 73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6 not found: ID does not exist" containerID="73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6"
May 26 11:48:54.899011 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.898984    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6"} err="failed to get container status \"73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6\": rpc error: code = NotFound desc = could not find container \"73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6\": container with ID starting with 73d2840764425cf1c3a052bfa5b495561dd909971bd238e33cc9ff57b7ad77a6 not found: ID does not exist"
May 26 11:48:54.902657 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:54.902637    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-51723-predictor-9f4779f89-p72pj"]
May 26 11:48:56.798544 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:48:56.798509    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" path="/var/lib/kubelet/pods/35c1c88b-4d98-42c3-86b5-2895bbb980fc/volumes"
May 26 11:49:20.450817 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.450780    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rpcs5/must-gather-n9nm9"]
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451141    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451158    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451168    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="storage-initializer"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451178    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="storage-initializer"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451186    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451194    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451204    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kube-rbac-proxy"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451212    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kube-rbac-proxy"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451235    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="storage-initializer"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451242    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="storage-initializer"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451255    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kube-rbac-proxy"
May 26 11:49:20.451288 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451262    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kube-rbac-proxy"
May 26 11:49:20.451865 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451322    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kube-rbac-proxy"
May 26 11:49:20.451865 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451334    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kube-rbac-proxy"
May 26 11:49:20.451865 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451345    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="4eb67ebf-1d83-4854-8077-8ed885de3d89" containerName="kserve-container"
May 26 11:49:20.451865 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.451355    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="35c1c88b-4d98-42c3-86b5-2895bbb980fc" containerName="kserve-container"
May 26 11:49:20.454169 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.454149    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.457088 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.457053    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpcs5\"/\"openshift-service-ca.crt\""
May 26 11:49:20.457190 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.457096    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpcs5\"/\"kube-root-ca.crt\""
May 26 11:49:20.457312 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.457296    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rpcs5\"/\"default-dockercfg-vrvc6\""
May 26 11:49:20.463541 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.463521    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpcs5/must-gather-n9nm9"]
May 26 11:49:20.475635 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.475617    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8465b5f1-e6f5-46ae-8748-988b92a5652b-must-gather-output\") pod \"must-gather-n9nm9\" (UID: \"8465b5f1-e6f5-46ae-8748-988b92a5652b\") " pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.475724 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.475641    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h62d\" (UniqueName: \"kubernetes.io/projected/8465b5f1-e6f5-46ae-8748-988b92a5652b-kube-api-access-7h62d\") pod \"must-gather-n9nm9\" (UID: \"8465b5f1-e6f5-46ae-8748-988b92a5652b\") " pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.576346 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.576324    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8465b5f1-e6f5-46ae-8748-988b92a5652b-must-gather-output\") pod \"must-gather-n9nm9\" (UID: \"8465b5f1-e6f5-46ae-8748-988b92a5652b\") " pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.576431 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.576350    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7h62d\" (UniqueName: \"kubernetes.io/projected/8465b5f1-e6f5-46ae-8748-988b92a5652b-kube-api-access-7h62d\") pod \"must-gather-n9nm9\" (UID: \"8465b5f1-e6f5-46ae-8748-988b92a5652b\") " pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.576658 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.576641    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8465b5f1-e6f5-46ae-8748-988b92a5652b-must-gather-output\") pod \"must-gather-n9nm9\" (UID: \"8465b5f1-e6f5-46ae-8748-988b92a5652b\") " pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.584419 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.584396    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h62d\" (UniqueName: \"kubernetes.io/projected/8465b5f1-e6f5-46ae-8748-988b92a5652b-kube-api-access-7h62d\") pod \"must-gather-n9nm9\" (UID: \"8465b5f1-e6f5-46ae-8748-988b92a5652b\") " pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.762528 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.762475    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpcs5/must-gather-n9nm9"
May 26 11:49:20.882730 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.882699    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpcs5/must-gather-n9nm9"]
May 26 11:49:20.885400 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:49:20.885366    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8465b5f1_e6f5_46ae_8748_988b92a5652b.slice/crio-5152a567ce9180e94c33b44800e79aca7c6e8d3a1da67d9390eae83e8366e0b3 WatchSource:0}: Error finding container 5152a567ce9180e94c33b44800e79aca7c6e8d3a1da67d9390eae83e8366e0b3: Status 404 returned error can't find the container with id 5152a567ce9180e94c33b44800e79aca7c6e8d3a1da67d9390eae83e8366e0b3
May 26 11:49:20.953044 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:20.953016    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpcs5/must-gather-n9nm9" event={"ID":"8465b5f1-e6f5-46ae-8748-988b92a5652b","Type":"ContainerStarted","Data":"5152a567ce9180e94c33b44800e79aca7c6e8d3a1da67d9390eae83e8366e0b3"}
May 26 11:49:21.960123 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:21.959261    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpcs5/must-gather-n9nm9" event={"ID":"8465b5f1-e6f5-46ae-8748-988b92a5652b","Type":"ContainerStarted","Data":"7fc9ff4bdd07a0bdaf22036089de04815cd70c43098e0c2728519188f24cfa38"}
May 26 11:49:21.960123 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:21.959310    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpcs5/must-gather-n9nm9" event={"ID":"8465b5f1-e6f5-46ae-8748-988b92a5652b","Type":"ContainerStarted","Data":"d6ea293b13607c20a4382dd94060096b1eee92cef4e67664025657beff0d4bab"}
May 26 11:49:21.976991 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:21.976935    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-rpcs5/must-gather-n9nm9" podStartSLOduration=1.2944011309999999 podStartE2EDuration="1.976917921s" podCreationTimestamp="2026-05-26 11:49:20 +0000 UTC" firstStartedPulling="2026-05-26 11:49:20.887153297 +0000 UTC m=+1482.709537500" lastFinishedPulling="2026-05-26 11:49:21.569670081 +0000 UTC m=+1483.392054290" observedRunningTime="2026-05-26 11:49:21.974704495 +0000 UTC m=+1483.797088721" watchObservedRunningTime="2026-05-26 11:49:21.976917921 +0000 UTC m=+1483.799302150"
May 26 11:49:23.016987 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:23.016954    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wndk5_272e6436-bf5b-4dbc-bd5b-c0b2baf62047/global-pull-secret-syncer/0.log"
May 26 11:49:23.062130 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:23.061990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-chvcq_cd00228a-356f-4547-b466-403f1cedb343/konnectivity-agent/0.log"
May 26 11:49:23.156501 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:23.156471    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-203.ec2.internal_691b2498c3b1cc38487ba5d11d3c5965/haproxy/0.log"
May 26 11:49:26.712485 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:26.712456    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkwcq_76681724-2bcb-45ed-9408-973e8969c50a/node-exporter/0.log"
May 26 11:49:26.735452 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:26.735420    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkwcq_76681724-2bcb-45ed-9408-973e8969c50a/kube-rbac-proxy/0.log"
May 26 11:49:26.760721 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:26.760695    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkwcq_76681724-2bcb-45ed-9408-973e8969c50a/init-textfile/0.log"
May 26 11:49:28.798021 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:28.797990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-845q9_61089709-9697-43cb-bf80-a0d676d4511c/networking-console-plugin/0.log"
May 26 11:49:30.077813 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.077780    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"]
May 26 11:49:30.084024 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.083997    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.089669 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.089643    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"]
May 26 11:49:30.158046 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.158014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qfk\" (UniqueName: \"kubernetes.io/projected/18efff6d-1762-4b06-8e85-a07f1f27215c-kube-api-access-x4qfk\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.158234 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.158055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-proc\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.158234 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.158096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-sys\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.158234 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.158200    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-lib-modules\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.158353 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.158290    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-podres\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259163 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259128    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-lib-modules\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259336 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259188    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-podres\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259336 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259244    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qfk\" (UniqueName: \"kubernetes.io/projected/18efff6d-1762-4b06-8e85-a07f1f27215c-kube-api-access-x4qfk\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259336 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259274    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-proc\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259336 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259297    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-sys\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259336 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259306    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-lib-modules\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259623 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259356    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-podres\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259623 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259367    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-proc\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.259623 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.259366    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18efff6d-1762-4b06-8e85-a07f1f27215c-sys\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.269282 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.269257    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qfk\" (UniqueName: \"kubernetes.io/projected/18efff6d-1762-4b06-8e85-a07f1f27215c-kube-api-access-x4qfk\") pod \"perf-node-gather-daemonset-bmj7s\" (UID: \"18efff6d-1762-4b06-8e85-a07f1f27215c\") " pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.396165 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.396090    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:30.540484 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.540441    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"]
May 26 11:49:30.544297 ip-10-0-132-203 kubenswrapper[2557]: W0526 11:49:30.544264    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod18efff6d_1762_4b06_8e85_a07f1f27215c.slice/crio-56af55ef3fe6f3e7e6647cd700ad2b64e37c0cbac273d40b7791dd021f8f108f WatchSource:0}: Error finding container 56af55ef3fe6f3e7e6647cd700ad2b64e37c0cbac273d40b7791dd021f8f108f: Status 404 returned error can't find the container with id 56af55ef3fe6f3e7e6647cd700ad2b64e37c0cbac273d40b7791dd021f8f108f
May 26 11:49:30.545966 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.545948    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:49:30.991486 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.991400    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s" event={"ID":"18efff6d-1762-4b06-8e85-a07f1f27215c","Type":"ContainerStarted","Data":"3c22cded5d68fdaa0e239aea55c84105b2974d4b5fd5fa32a217b90d550caa47"}
May 26 11:49:30.991486 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.991436    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s" event={"ID":"18efff6d-1762-4b06-8e85-a07f1f27215c","Type":"ContainerStarted","Data":"56af55ef3fe6f3e7e6647cd700ad2b64e37c0cbac273d40b7791dd021f8f108f"}
May 26 11:49:30.991707 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:30.991511    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:31.011532 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:31.011512    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fbqd_89d04d62-aed1-4b37-93fe-170fafdb6e2d/dns/0.log"
May 26 11:49:31.020438 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:31.020405    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s" podStartSLOduration=1.020373094 podStartE2EDuration="1.020373094s" podCreationTimestamp="2026-05-26 11:49:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:49:31.020284245 +0000 UTC m=+1492.842668471" watchObservedRunningTime="2026-05-26 11:49:31.020373094 +0000 UTC m=+1492.842757318"
May 26 11:49:31.069739 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:31.069711    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fbqd_89d04d62-aed1-4b37-93fe-170fafdb6e2d/kube-rbac-proxy/0.log"
May 26 11:49:31.230359 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:31.230335    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q98hj_ae5a3ccd-890a-464e-a273-c2a2550652f8/dns-node-resolver/0.log"
May 26 11:49:31.709556 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:31.709500    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5gk8_ec1fce1f-f871-4081-8829-c038c14b7fea/node-ca/0.log"
May 26 11:49:33.048391 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:33.048357    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9v24s_f9e2f963-efe8-4935-987d-0661094ac33f/serve-healthcheck-canary/0.log"
May 26 11:49:33.610329 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:33.610298    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t6gkb_257582b4-4b4d-4ab5-b7ef-0350b45efad9/kube-rbac-proxy/0.log"
May 26 11:49:33.630953 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:33.630928    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t6gkb_257582b4-4b4d-4ab5-b7ef-0350b45efad9/exporter/0.log"
May 26 11:49:33.654262 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:33.654233    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t6gkb_257582b4-4b4d-4ab5-b7ef-0350b45efad9/extractor/0.log"
May 26 11:49:35.693565 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:35.693538    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-w4h84_0c66c557-047c-4540-b226-6b0831e097f2/server/0.log"
May 26 11:49:35.842309 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:35.842269    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-2mh8b_f8f40655-494c-44f3-9359-5cbbb77ddbcb/manager/0.log"
May 26 11:49:35.889909 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:35.889870    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-x6rsx_ab42601b-d8ff-4e76-afb7-7029cfabac7d/seaweedfs/0.log"
May 26 11:49:37.003492 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:37.003457    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rpcs5/perf-node-gather-daemonset-bmj7s"
May 26 11:49:40.661391 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:40.661361    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:49:40.666141 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:40.666115    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:49:41.731919 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.731890    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wkgxh_08fa3909-7685-4b83-8d8b-53f0016a570b/kube-multus-additional-cni-plugins/0.log"
May 26 11:49:41.757004 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.756979    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wkgxh_08fa3909-7685-4b83-8d8b-53f0016a570b/egress-router-binary-copy/0.log"
May 26 11:49:41.785694 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.785644    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wkgxh_08fa3909-7685-4b83-8d8b-53f0016a570b/cni-plugins/0.log"
May 26 11:49:41.808413 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.808391    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wkgxh_08fa3909-7685-4b83-8d8b-53f0016a570b/bond-cni-plugin/0.log"
May 26 11:49:41.830780 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.830758    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wkgxh_08fa3909-7685-4b83-8d8b-53f0016a570b/routeoverride-cni/0.log"
May 26 11:49:41.851763 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.851711    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wkgxh_08fa3909-7685-4b83-8d8b-53f0016a570b/whereabouts-cni-bincopy/0.log"
May 26 11:49:41.870821 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.870806    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wkgxh_08fa3909-7685-4b83-8d8b-53f0016a570b/whereabouts-cni/0.log"
May 26 11:49:41.903859 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:41.903840    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dq4tc_f610ed13-1463-4dc3-9769-6f6876ae0716/kube-multus/0.log"
May 26 11:49:42.086056 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:42.086029    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t24mf_d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20/network-metrics-daemon/0.log"
May 26 11:49:42.112852 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:42.112798    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t24mf_d9e2271b-ccaf-4a22-a1ba-8bc9f6479b20/kube-rbac-proxy/0.log"
May 26 11:49:43.639200 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.639163    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-controller/0.log"
May 26 11:49:43.657783 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.657756    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/0.log"
May 26 11:49:43.671756 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.671730    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovn-acl-logging/1.log"
May 26 11:49:43.697161 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.697145    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/kube-rbac-proxy-node/0.log"
May 26 11:49:43.721876 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.721852    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/kube-rbac-proxy-ovn-metrics/0.log"
May 26 11:49:43.748667 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.748642    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/northd/0.log"
May 26 11:49:43.770050 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.770035    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/nbdb/0.log"
May 26 11:49:43.793294 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.793270    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/sbdb/0.log"
May 26 11:49:43.957111 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:43.957035    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zdkwm_deeb388a-9cbe-4327-bfd2-96b184f74d16/ovnkube-controller/0.log"
May 26 11:49:45.032110 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:45.032061    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-m5xdx_5763dd28-8ea6-484b-a856-94899f1552dc/network-check-target-container/0.log"
May 26 11:49:46.044883 ip-10-0-132-203 kubenswrapper[2557]: I0526 11:49:46.044858    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-596zc_e4bee2d0-150b-4867-b79e-e5a0d7beb916/iptables-alerter/0.log"