May 25 11:17:21.122005 ip-10-0-143-48 systemd[1]: Starting Kubernetes Kubelet...
May 25 11:17:21.585983 ip-10-0-143-48 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:21.585983 ip-10-0-143-48 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 11:17:21.585983 ip-10-0-143-48 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:21.585983 ip-10-0-143-48 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 11:17:21.585983 ip-10-0-143-48 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:21.586861 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.586772    2566 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 11:17:21.591858 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591836    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:21.591858 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591858    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591862    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591865    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591868    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591871    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591874    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591878    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591883    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591886    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591889    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591892    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591895    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591898    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591901    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591903    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591906    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591909    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591911    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591914    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591917    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:21.591940 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591920    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591922    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591925    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591934    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591937    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591940    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591942    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591945    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591948    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591951    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591954    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591957    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591959    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591962    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591964    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591967    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591970    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591972    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591975    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591978    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:21.592416 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591980    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591983    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591986    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591989    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591991    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591994    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.591997    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592001    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592005    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592008    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592012    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592015    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592018    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592020    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592024    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592027    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592030    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592032    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592035    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:21.592959 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592038    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592040    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592043    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592046    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592048    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592051    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592053    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592056    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592059    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592061    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592064    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592067    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592069    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592072    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592074    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592078    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592080    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592083    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592086    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592088    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:21.593421 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592091    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592094    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592096    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592099    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592101    2566 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.592107    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593116    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593123    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593126    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593130    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593132    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593136    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593138    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593141    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593145    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593147    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593150    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593153    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593155    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593158    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:21.593902 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593161    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593164    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593166    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593169    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593171    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593174    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593177    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593180    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593182    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593185    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593188    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593190    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593193    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593195    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593198    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593201    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593203    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593206    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593209    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593211    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:21.594400 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593214    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593216    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593219    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593221    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593224    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593226    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593232    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593236    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593239    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593241    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593244    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593246    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593249    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593251    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593254    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593256    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593259    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593262    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593264    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593267    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:21.594936 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593270    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593272    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593275    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593278    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593280    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593283    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593285    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593288    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593290    2566 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593293    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593295    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593299    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593302    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593305    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593308    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593311    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593314    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593316    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593320    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593323    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:21.595444 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593325    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593328    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593331    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593334    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593337    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593339    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593342    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593344    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593346    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593349    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593352    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.593354    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593431    2566 flags.go:64] FLAG: --address="0.0.0.0"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593444    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593450    2566 flags.go:64] FLAG: --anonymous-auth="true"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593455    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593460    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593464    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593468    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593476    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 11:17:21.595957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593479    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593483    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593486    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593490    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593493    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593496    2566 flags.go:64] FLAG: --cgroup-root=""
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593499    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593502    2566 flags.go:64] FLAG: --client-ca-file=""
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593505    2566 flags.go:64] FLAG: --cloud-config=""
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593508    2566 flags.go:64] FLAG: --cloud-provider="external"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593511    2566 flags.go:64] FLAG: --cluster-dns="[]"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593516    2566 flags.go:64] FLAG: --cluster-domain=""
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593519    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593522    2566 flags.go:64] FLAG: --config-dir=""
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593525    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593528    2566 flags.go:64] FLAG: --container-log-max-files="5"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593533    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593548    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593552    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593555    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593559    2566 flags.go:64] FLAG: --contention-profiling="false"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593562    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593565    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593568    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593571    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 11:17:21.596438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593576    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593580    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593583    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593586    2566 flags.go:64] FLAG: --enable-load-reader="false"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593589    2566 flags.go:64] FLAG: --enable-server="true"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593592    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593597    2566 flags.go:64] FLAG: --event-burst="100"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593600    2566 flags.go:64] FLAG: --event-qps="50"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593603    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593606    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593609    2566 flags.go:64] FLAG: --eviction-hard=""
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593613    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593616    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593619    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593622    2566 flags.go:64] FLAG: --eviction-soft=""
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593625    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593628    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593632    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593635    2566 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593638    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593641    2566 flags.go:64] FLAG: --fail-swap-on="true"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593644    2566 flags.go:64] FLAG: --feature-gates=""
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593648    2566 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593651    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593654    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 11:17:21.597067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593657    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593660    2566 flags.go:64] FLAG: --healthz-port="10248"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593663    2566 flags.go:64] FLAG: --help="false"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593666    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-143-48.ec2.internal"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593669    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593672    2566 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593675    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593679    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593682    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593687    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593690    2566 flags.go:64] FLAG: --image-service-endpoint=""
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593695    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593698    2566 flags.go:64] FLAG: --kube-api-burst="100"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593702    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593705    2566 flags.go:64] FLAG: --kube-api-qps="50"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593708    2566 flags.go:64] FLAG: --kube-reserved=""
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593711    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593714    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593717    2566 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593720    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593723    2566 flags.go:64] FLAG: --lock-file=""
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593726    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593729    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593732    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 11:17:21.597692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593738    2566 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593741    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593744    2566 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593747    2566 flags.go:64] FLAG: --logging-format="text"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593750    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593754    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593757    2566 flags.go:64] FLAG: --manifest-url=""
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593759    2566 flags.go:64] FLAG: --manifest-url-header=""
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593764    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593767    2566 flags.go:64] FLAG: --max-open-files="1000000"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593771    2566 flags.go:64] FLAG: --max-pods="110"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593774    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593778    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593781    2566 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593784    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593787    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593790    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593793    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593802    2566 flags.go:64] FLAG: --node-status-max-images="50"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593807    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593810    2566 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593813    2566 flags.go:64] FLAG: --pod-cidr=""
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593816    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 11:17:21.598259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593822    2566 flags.go:64] FLAG: --pod-manifest-path=""
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593825    2566 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593828    2566 flags.go:64] FLAG: --pods-per-core="0"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593832    2566 flags.go:64] FLAG: --port="10250"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593835    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593838    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06d48630e0aeb4c36"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593841    2566 flags.go:64] FLAG: --qos-reserved=""
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593845    2566 flags.go:64] FLAG: --read-only-port="10255"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593848    2566 flags.go:64] FLAG: --register-node="true"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593851    2566 flags.go:64] FLAG: --register-schedulable="true"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593854    2566 flags.go:64] FLAG: --register-with-taints=""
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593857    2566 flags.go:64] FLAG: --registry-burst="10"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593860    2566 flags.go:64] FLAG: --registry-qps="5"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593863    2566 flags.go:64] FLAG: --reserved-cpus=""
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593866    2566 flags.go:64] FLAG: --reserved-memory=""
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593870    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593873    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593876    2566 flags.go:64] FLAG: --rotate-certificates="false"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593879    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593882    2566 flags.go:64] FLAG: --runonce="false"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593885    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593888    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593892    2566 flags.go:64] FLAG: --seccomp-default="false"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593894    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593897    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593901    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 11:17:21.598874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593904    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593907    2566 flags.go:64] FLAG: --storage-driver-password="root"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593912    2566 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593916    2566 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593919    2566 flags.go:64] FLAG: --storage-driver-user="root"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593922    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593925    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593928    2566 flags.go:64] FLAG: --system-cgroups=""
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593931    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593936    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593940    2566 flags.go:64] FLAG: --tls-cert-file=""
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593942    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593947    2566 flags.go:64] FLAG: --tls-min-version=""
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593950    2566 flags.go:64] FLAG: --tls-private-key-file=""
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593953    2566 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593956    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593959    2566 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593962    2566 flags.go:64] FLAG: --v="2"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593966    2566 flags.go:64] FLAG: --version="false"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593971    2566 flags.go:64] FLAG: --vmodule=""
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593975    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.593979    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594077    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594081    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:21.599557 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594084    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594087    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594090    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594096    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594099    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594101    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594104    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594107    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594110    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594112    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594115    2566 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594119    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594121    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594124    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594127    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594130    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594132    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594135    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594137    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594140    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:21.600237 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594143    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594146    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594148    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594151    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594153    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594156    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594159    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594161    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594164    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594166    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594169    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594173    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594177    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594179    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594182    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594185    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594188    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594191    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594193    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:21.600803 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594196    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594199    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594201    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594204    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594207    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594211    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594213    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594216    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594219    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594221    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594224    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594226    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594229    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594232    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594234    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594237    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594240    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594242    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594245    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594247    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:21.601281 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594251    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594254    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594258    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594260    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594263    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594266    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594269    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594271    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594275    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594278    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594280    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594283    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594285    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594288    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594291    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594294    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594297    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594299    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594303    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:21.601797 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594305    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594308    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594310    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594313    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594315    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.594318    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.594323    2566 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.601535    2566 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.601565    2566 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601616    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601621    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601624    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601628    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601630    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601633    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:21.602252 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601638    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601642    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601645    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601648    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601651    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601654    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601657    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601660    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601662    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601665    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601668    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601670    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601673    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601675    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601679    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601681    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601685    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601688    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601691    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601693    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:21.602652 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601696    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601698    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601701    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601703    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601706    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601709    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601712    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601715    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601718    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601721    2566 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601723    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601726    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601728    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601732    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601736    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601738    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601741    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601744    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601747    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601749    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:21.603137 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601752    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601754    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601757    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601759    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601762    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601765    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601767    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601770    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601773    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601777    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601780    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601782    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601785    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601787    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601790    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601793    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601796    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601799    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601803    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:21.603683 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601805    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601808    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601811    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601814    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601817    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601819    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601822    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601825    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601828    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601830    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601833    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601836    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601838    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601841    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601843    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601846    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601848    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601851    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601855    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601857    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:21.604190 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601860    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.601865    2566 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601970    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601975    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601979    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601981    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601984    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601987    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601989    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601992    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601996    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.601999    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602001    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602004    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602007    2566 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602010    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:21.604700 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602012    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602015    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602017    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602020    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602023    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602025    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602028    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602031    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602033    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602036    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602038    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602041    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602043    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602046    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602048    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602051    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602054    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602056    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602058    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602061    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:21.605103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602064    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602067    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602069    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602072    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602074    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602077    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602080    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602083    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602085    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602088    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602090    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602093    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602096    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602098    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602101    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602103    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602107    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602111    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602114    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:21.605600 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602117    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602119    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602122    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602125    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602128    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602130    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602133    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602135    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602137    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602140    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602143    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602145    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602148    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602150    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602155    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602158    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602161    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602164    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602166    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:21.606083 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602169    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602173    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602176    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602179    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602182    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602185    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602187    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602190    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602192    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602195    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602198    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602200    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602203    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:21.602205    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.602210    2566 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:21.606567 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.603056    2566 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 11:17:21.606945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.605339    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 11:17:21.606945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.606500    2566 server.go:1019] "Starting client certificate rotation"
May 25 11:17:21.606945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.606606    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:21.606945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.606656    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:21.635620 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.635597    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:21.637366 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.637326    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:21.655665 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.655638    2566 log.go:25] "Validated CRI v1 runtime API"
May 25 11:17:21.660948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.660923    2566 log.go:25] "Validated CRI v1 image API"
May 25 11:17:21.663123 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.663100    2566 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 11:17:21.667510 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.667489    2566 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:21.670669 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.670641    2566 fs.go:135] Filesystem UUIDs: map[487c0a1c-6258-441a-9b71-dc4a2152cebe:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9d18d79f-aaeb-4381-a8e7-24e5a3f0d484:/dev/nvme0n1p3]
May 25 11:17:21.670726 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.670670    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 11:17:21.675100 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.674986    2566 manager.go:217] Machine: {Timestamp:2026-05-25 11:17:21.674468236 +0000 UTC m=+0.429445038 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095310 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c93b156dee2e58743f4a6e94f9b88 SystemUUID:ec2c93b1-56de-e2e5-8743-f4a6e94f9b88 BootID:8efe0290-1ac6-45c4-b661-66271ab06473 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:1a:07:f9:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:1a:07:f9:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:f7:8b:b1:b9:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 11:17:21.675100 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.675097    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 11:17:21.675226 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.675179    2566 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 11:17:21.676951 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.676922    2566 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 11:17:21.677099 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.676952    2566 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-48.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 11:17:21.677184 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.677113    2566 topology_manager.go:138] "Creating topology manager with none policy"
May 25 11:17:21.677184 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.677122    2566 container_manager_linux.go:306] "Creating device plugin manager"
May 25 11:17:21.677184 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.677130    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:21.677184 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.677141    2566 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:21.679484 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.679469    2566 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:21.679632 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.679622    2566 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 11:17:21.682003 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.681991    2566 kubelet.go:491] "Attempting to sync node with API server"
May 25 11:17:21.682053 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.682011    2566 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 11:17:21.682053 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.682025    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 11:17:21.682053 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.682039    2566 kubelet.go:397] "Adding apiserver pod source"
May 25 11:17:21.682053 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.682048    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 11:17:21.683168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.683154    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:21.683243 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.683174    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:21.686270 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.686252    2566 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 11:17:21.688125 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.688102    2566 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 11:17:21.689210 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689195    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 11:17:21.689292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689217    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 11:17:21.689292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689226    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 11:17:21.689292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689233    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 11:17:21.689292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689246    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 11:17:21.689292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689256    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 11:17:21.689292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689264    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 11:17:21.689292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689273    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 11:17:21.689564 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689315    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 11:17:21.689564 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689325    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 11:17:21.689564 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689346    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 11:17:21.689564 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.689492    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 11:17:21.690922 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.690907    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 11:17:21.690997 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.690925    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 11:17:21.694332 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.694313    2566 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 11:17:21.694426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.694385    2566 server.go:1295] "Started kubelet"
May 25 11:17:21.694620 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.694559    2566 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 11:17:21.694720 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.694640    2566 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 11:17:21.695253 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.695220    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 11:17:21.695318 ip-10-0-143-48 systemd[1]: Started Kubernetes Kubelet.
May 25 11:17:21.696001 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.695983    2566 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 11:17:21.699043 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.698998    2566 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 11:17:21.699127 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.699044    2566 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 11:17:21.699327 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.699299    2566 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 11:17:21.701218 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.701199    2566 server.go:317] "Adding debug handlers to kubelet server"
May 25 11:17:21.704249 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.704211    2566 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 11:17:21.704691 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.704668    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 11:17:21.704691 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.704684    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 11:17:21.704841 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.704742    2566 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 11:17:21.704841 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.704751    2566 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 11:17:21.704841 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.704800    2566 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 11:17:21.704994 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.704905    2566 reconstruct.go:97] "Volume reconstruction finished"
May 25 11:17:21.704994 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.704914    2566 reconciler.go:26] "Reconciler: start to sync state"
May 25 11:17:21.705151 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.704202    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18b2cada4c2ae795  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-05-25 11:17:21.694328725 +0000 UTC m=+0.449305517,LastTimestamp:2026-05-25 11:17:21.694328725 +0000 UTC m=+0.449305517,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
May 25 11:17:21.705794 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.705619    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:21.705794 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.705635    2566 factory.go:55] Registering systemd factory
May 25 11:17:21.705794 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.705670    2566 factory.go:223] Registration of the systemd container factory successfully
May 25 11:17:21.705991 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.705908    2566 factory.go:153] Registering CRI-O factory
May 25 11:17:21.705991 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.705926    2566 factory.go:223] Registration of the crio container factory successfully
May 25 11:17:21.705991 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.705982    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 11:17:21.706246 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.706002    2566 factory.go:103] Registering Raw factory
May 25 11:17:21.706246 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.706017    2566 manager.go:1196] Started watching for new ooms in manager
May 25 11:17:21.706704 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.706690    2566 manager.go:319] Starting recovery of all containers
May 25 11:17:21.716235 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.716085    2566 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 11:17:21.716235 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.716206    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 11:17:21.716673 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.716577    2566 manager.go:324] Recovery completed
May 25 11:17:21.721499 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.721485    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:21.724571 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.723973    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:21.724571 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.724184    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q29j6"
May 25 11:17:21.724571 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.724235    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:21.724571 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.724266    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:21.725387 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.725369    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 11:17:21.725387 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.725385    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 11:17:21.725480 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.725402    2566 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:21.726485 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.726418    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18b2cada4defc6c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-05-25 11:17:21.72400813 +0000 UTC m=+0.478984917,LastTimestamp:2026-05-25 11:17:21.72400813 +0000 UTC m=+0.478984917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
May 25 11:17:21.727808 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.727796    2566 policy_none.go:49] "None policy: Start"
May 25 11:17:21.727855 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.727812    2566 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 11:17:21.727855 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.727824    2566 state_mem.go:35] "Initializing new in-memory state store"
May 25 11:17:21.731051 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.731029    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q29j6"
May 25 11:17:21.766227 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.766205    2566 manager.go:341] "Starting Device Plugin manager"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.766248    2566 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.766262    2566 server.go:85] "Starting device plugin registration server"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.766525    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.766535    2566 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.766642    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.766749    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.766775    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.767340    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 11:17:21.774198 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.767373    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:21.829258 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.829218    2566 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 11:17:21.830447 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.830433    2566 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 11:17:21.830513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.830454    2566 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 11:17:21.830513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.830473    2566 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 11:17:21.830513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.830480    2566 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 11:17:21.830662 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.830525    2566 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 11:17:21.837618 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.837524    2566 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:21.867099 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.867071    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:21.869468 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.869451    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:21.869549 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.869483    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:21.869549 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.869494    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:21.869549 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.869516    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-48.ec2.internal"
May 25 11:17:21.879431 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.879414    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-48.ec2.internal"
May 25 11:17:21.879477 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.879439    2566 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-48.ec2.internal\": node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:21.897204 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.897180    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:21.931035 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.931002    2566 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"]
May 25 11:17:21.931176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.931081    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:21.932466 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.932450    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:21.932582 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.932476    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:21.932582 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.932487    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:21.933657 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.933644    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:21.933880 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.933839    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:21.933880 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.933877    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:21.934422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.934405    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:21.934509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.934432    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:21.934509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.934441    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:21.934509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.934408    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:21.934645 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.934532    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:21.934645 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.934558    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:21.935626 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.935611    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
May 25 11:17:21.935687 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.935637    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:21.936304 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.936287    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:21.936379 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.936313    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:21.936379 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:21.936322    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:21.959363 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.959337    2566 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-48.ec2.internal\" not found" node="ip-10-0-143-48.ec2.internal"
May 25 11:17:21.963727 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.963710    2566 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-48.ec2.internal\" not found" node="ip-10-0-143-48.ec2.internal"
May 25 11:17:21.998010 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:21.997977    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.007348 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.007320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/618d39cc139f57474525689dc86df07c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"618d39cc139f57474525689dc86df07c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.007474 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.007351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/618d39cc139f57474525689dc86df07c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"618d39cc139f57474525689dc86df07c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.007474 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.007373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/324b37d0baef0cf5218e4c8fcb924549-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.ec2.internal\" (UID: \"324b37d0baef0cf5218e4c8fcb924549\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.098383 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.098355    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.107771 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.107739    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/618d39cc139f57474525689dc86df07c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"618d39cc139f57474525689dc86df07c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.107906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.107776    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/618d39cc139f57474525689dc86df07c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"618d39cc139f57474525689dc86df07c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.107906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.107796    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/324b37d0baef0cf5218e4c8fcb924549-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.ec2.internal\" (UID: \"324b37d0baef0cf5218e4c8fcb924549\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.107906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.107843    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/324b37d0baef0cf5218e4c8fcb924549-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.ec2.internal\" (UID: \"324b37d0baef0cf5218e4c8fcb924549\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.107906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.107848    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/618d39cc139f57474525689dc86df07c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"618d39cc139f57474525689dc86df07c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.107906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.107853    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/618d39cc139f57474525689dc86df07c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"618d39cc139f57474525689dc86df07c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.199201 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.199148    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.262652 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.262614    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.266393 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.266370    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
May 25 11:17:22.300214 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.300181    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.400757 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.400680    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.501200 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.501178    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.601837 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.601808    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.606056 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.606037    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 11:17:22.606211 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.606196    2566 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:22.702614 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.702526    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.705207 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.705189    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 11:17:22.717499 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.717469    2566 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:22.733238 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.733193    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:12:21 +0000 UTC" deadline="2027-12-05 12:40:53.246147901 +0000 UTC"
May 25 11:17:22.733238 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.733235    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13417h23m30.512918899s"
May 25 11:17:22.747775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.747748    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2vb78"
May 25 11:17:22.751747 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:22.751718    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618d39cc139f57474525689dc86df07c.slice/crio-a7dff5f03c98257d9d57f1d88ff4e0d38c8416021fff8d6c1190f89e019bcbc9 WatchSource:0}: Error finding container a7dff5f03c98257d9d57f1d88ff4e0d38c8416021fff8d6c1190f89e019bcbc9: Status 404 returned error can't find the container with id a7dff5f03c98257d9d57f1d88ff4e0d38c8416021fff8d6c1190f89e019bcbc9
May 25 11:17:22.752027 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:22.752005    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324b37d0baef0cf5218e4c8fcb924549.slice/crio-8afdbc33501117e6923d0da7212b395cb280cf252127beef54d452634b474c3f WatchSource:0}: Error finding container 8afdbc33501117e6923d0da7212b395cb280cf252127beef54d452634b474c3f: Status 404 returned error can't find the container with id 8afdbc33501117e6923d0da7212b395cb280cf252127beef54d452634b474c3f
May 25 11:17:22.756653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.756636    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2vb78"
May 25 11:17:22.756841 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.756825    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:17:22.802668 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.802620    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.835065 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.835007    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal" event={"ID":"324b37d0baef0cf5218e4c8fcb924549","Type":"ContainerStarted","Data":"8afdbc33501117e6923d0da7212b395cb280cf252127beef54d452634b474c3f"}
May 25 11:17:22.835977 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.835955    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" event={"ID":"618d39cc139f57474525689dc86df07c","Type":"ContainerStarted","Data":"a7dff5f03c98257d9d57f1d88ff4e0d38c8416021fff8d6c1190f89e019bcbc9"}
May 25 11:17:22.903265 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:22.903230    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:22.926150 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.926123    2566 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:22.928315 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:22.928294    2566 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:23.004063 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.003992    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:23.104448 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.104412    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:23.205221 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.205179    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
May 25 11:17:23.238110 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.237946    2566 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:23.305011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.304846    2566 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
May 25 11:17:23.319813 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.319696    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:23.321071 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.321042    2566 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
May 25 11:17:23.331266 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.331240    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:23.683511 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.683463    2566 apiserver.go:52] "Watching apiserver"
May 25 11:17:23.691336 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.691305    2566 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 11:17:23.694846 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.694818    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-jqzls","openshift-image-registry/node-ca-z7trl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-6fxqg","kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn","openshift-cluster-node-tuning-operator/tuned-rk6gc","openshift-multus/multus-additional-cni-plugins-5f87w","openshift-multus/multus-mm888","openshift-multus/network-metrics-daemon-xm9qk","openshift-network-diagnostics/network-check-target-dcdqx","openshift-network-operator/iptables-alerter-7kjd5","kube-system/konnectivity-agent-6mk6l"]
May 25 11:17:23.697486 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.697459    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.699612 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.699528    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.699892 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.699806    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l6vpk\""
May 25 11:17:23.699892 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.699880    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 11:17:23.700035 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.699989    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:23.701528 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.701506    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:23.701682 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.701641    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:23.701682 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.701673    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 11:17:23.702136 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.702084    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 11:17:23.702136 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.702122    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 11:17:23.702279 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.702171    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 11:17:23.702728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.702627    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 11:17:23.702728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.702640    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 11:17:23.702728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.702718    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdwwv\""
May 25 11:17:23.703743 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.703720    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.705643 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.705624    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pmns6\""
May 25 11:17:23.705643 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.705637    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 11:17:23.705910 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.705894    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 11:17:23.706023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.705949    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:23.706268 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.706249    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.708135 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.708117    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 11:17:23.708228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.708167    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 11:17:23.708291 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.708242    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2hp4h\""
May 25 11:17:23.708526 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.708509    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 11:17:23.710639 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.710605    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.710919 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.710900    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.713787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.713052    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl6ph\""
May 25 11:17:23.713787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.713174    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 11:17:23.713787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.713253    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.713787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.713393    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 11:17:23.713787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.713454    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 11:17:23.713787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.713689    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 11:17:23.714368 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.714159    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 11:17:23.714368 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.714267    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xnrpf\""
May 25 11:17:23.715751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.715735    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 11:17:23.715850 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.715781    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b9qpb\""
May 25 11:17:23.716262 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.716229    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 11:17:23.716510 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.716485    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mm888"
May 25 11:17:23.717014 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.716993    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 11:17:23.717138 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.717114    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 11:17:23.717753 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.717418    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 11:17:23.717753 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.717666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.717753 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.717710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-node-log\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.717753 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.717753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysconfig\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.718007 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.717819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-tuned\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.718071 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-run-netns\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.718357 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-systemd\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.718357 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcrn7\" (UniqueName: \"kubernetes.io/projected/4946c3b1-5fd0-4cb9-80b5-48a624edd688-kube-api-access-jcrn7\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.718357 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-modprobe-d\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.718357 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysctl-d\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.718357 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-var-lib-kubelet\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.718650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-kubelet\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.718650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718414    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-log-socket\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.718650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovnkube-script-lib\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.718941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovn-node-metrics-cert\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-kubernetes\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719074    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysctl-conf\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719214    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-df4fj\""
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-run\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-sys\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-lib-modules\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-systemd-units\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8ls\" (UniqueName: \"kubernetes.io/projected/d14d47c1-872a-4915-9cc5-f57cf19c52bb-kube-api-access-kb8ls\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-ovn\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkjd\" (UniqueName: \"kubernetes.io/projected/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-kube-api-access-kqkjd\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719983    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.719963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-slash\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d14d47c1-872a-4915-9cc5-f57cf19c52bb-host\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.721034 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.720064    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-var-lib-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-cni-bin\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-cni-netd\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-env-overrides\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4946c3b1-5fd0-4cb9-80b5-48a624edd688-iptables-alerter-script\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4946c3b1-5fd0-4cb9-80b5-48a624edd688-host-slash\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-systemd\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-tmp\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d14d47c1-872a-4915-9cc5-f57cf19c52bb-serviceca\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.720992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-host\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.721020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-etc-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.721075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovnkube-config\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.721943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.721125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jf2\" (UniqueName: \"kubernetes.io/projected/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-kube-api-access-48jf2\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.723783 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.723762    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:23.726065 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.726046    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kzk9c\""
May 25 11:17:23.726163 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.726098    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 11:17:23.726223 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.726167    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 11:17:23.757757 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.757727    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:22 +0000 UTC" deadline="2028-02-17 10:02:32.530907074 +0000 UTC"
May 25 11:17:23.757757 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.757756    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15190h45m8.773154333s"
May 25 11:17:23.788770 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.788740    2566 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:23.805531 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.805501    2566 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 11:17:23.822325 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822292    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysctl-d\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822329    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-var-lib-kubelet\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822354    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-kubelet\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822376    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822433    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-kubelet\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822433    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-var-lib-kubelet\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-k8s-cni-cncf-io\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822442    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822474    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-run\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822498    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysctl-d\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822501    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-systemd-units\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-os-release\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822564    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-systemd-units\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-socket-dir-parent\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822604    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-run\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-hostroot\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822661    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-ovn\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822689    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-slash\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822714    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d14d47c1-872a-4915-9cc5-f57cf19c52bb-host\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72vh\" (UniqueName: \"kubernetes.io/projected/d863a6d6-a9ed-4c86-b88f-2541282968b3-kube-api-access-q72vh\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822746    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-ovn\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822767    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-slash\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822770    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-cni-bin\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822776    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d14d47c1-872a-4915-9cc5-f57cf19c52bb-host\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822818    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-cni-netd\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822816    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-cni-bin\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.822974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822864    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-cni-netd\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822836    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-env-overrides\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822905    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4946c3b1-5fd0-4cb9-80b5-48a624edd688-iptables-alerter-script\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822939    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4946c3b1-5fd0-4cb9-80b5-48a624edd688-host-slash\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.822990    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-tmp\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823017    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823038    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4946c3b1-5fd0-4cb9-80b5-48a624edd688-host-slash\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-conf-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42swz\" (UniqueName: \"kubernetes.io/projected/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-kube-api-access-42swz\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823205    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-host\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cnibin\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqb7\" (UniqueName: \"kubernetes.io/projected/e5d18530-e2b7-49bf-906a-2969cfb88e1d-kube-api-access-7sqb7\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxm6s\" (UniqueName: \"kubernetes.io/projected/7ad17068-4f7d-491f-8d77-61c903d241cb-kube-api-access-pxm6s\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823385    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823397    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-env-overrides\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823407    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-host\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.823778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823424    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-node-log\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823463    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysconfig\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823464    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823458    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823510    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jcrn7\" (UniqueName: \"kubernetes.io/projected/4946c3b1-5fd0-4cb9-80b5-48a624edd688-kube-api-access-jcrn7\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823525    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysconfig\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823534    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4946c3b1-5fd0-4cb9-80b5-48a624edd688-iptables-alerter-script\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823557    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-node-log\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-os-release\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9442f5d3-3e19-44ec-8db5-1706c9b11ba2-agent-certs\") pod \"konnectivity-agent-6mk6l\" (UID: \"9442f5d3-3e19-44ec-8db5-1706c9b11ba2\") " pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823732    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-modprobe-d\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823765    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-log-socket\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823789    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovnkube-script-lib\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823851    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-log-socket\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-socket-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-registration-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823927    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-modprobe-d\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.824611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.823975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824005    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824046    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824074    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovn-node-metrics-cert\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ad17068-4f7d-491f-8d77-61c903d241cb-hosts-file\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-daemon-config\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824288    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-kubernetes\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824345    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-kubernetes\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824366    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysctl-conf\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824397    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-sys\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824422    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-lib-modules\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824450    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8ls\" (UniqueName: \"kubernetes.io/projected/d14d47c1-872a-4915-9cc5-f57cf19c52bb-kube-api-access-kb8ls\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-system-cni-dir\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824495    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-sys\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-system-cni-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824520    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovnkube-script-lib\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.825376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824551    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-lib-modules\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824617    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-sysctl-conf\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-cni-multus\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824693    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkjd\" (UniqueName: \"kubernetes.io/projected/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-kube-api-access-kqkjd\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-kubelet\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-multus-certs\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824814    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-var-lib-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad17068-4f7d-491f-8d77-61c903d241cb-tmp-dir\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-etc-kubernetes\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9442f5d3-3e19-44ec-8db5-1706c9b11ba2-konnectivity-ca\") pod \"konnectivity-agent-6mk6l\" (UID: \"9442f5d3-3e19-44ec-8db5-1706c9b11ba2\") " pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824935    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-var-lib-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.824971    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-systemd\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825041    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d14d47c1-872a-4915-9cc5-f57cf19c52bb-serviceca\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825044    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-systemd\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825114    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-sys-fs\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-cni-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-cnibin\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-cni-binary-copy\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825282    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-etc-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825309    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovnkube-config\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825346    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-48jf2\" (UniqueName: \"kubernetes.io/projected/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-kube-api-access-48jf2\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825385    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-etc-openvswitch\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-device-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825468    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d14d47c1-872a-4915-9cc5-f57cf19c52bb-serviceca\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825561    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-tuned\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825607    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-run-netns\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825684    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-systemd\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-netns\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825748    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-host-run-netns\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-cni-bin\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825813    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-run-systemd\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.826784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.825824    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl7dc\" (UniqueName: \"kubernetes.io/projected/61545c61-84b8-432a-9443-e9fd8b59060f-kube-api-access-bl7dc\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:23.827504 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.827094    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-tmp\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.827504 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.827351    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovn-node-metrics-cert\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.827504 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.827409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-ovnkube-config\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.831672 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.831556    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-etc-tuned\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.832131 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.832110    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:23.832248 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.832135    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:23.832248 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.832151    2566 projected.go:194] Error preparing data for projected volume kube-api-access-zcwvm for pod openshift-network-diagnostics/network-check-target-dcdqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:23.832248 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.832225    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm podName:2397ea2a-dfea-45e4-8858-e51400496bff nodeName:}" failed. No retries permitted until 2026-05-25 11:17:24.332205586 +0000 UTC m=+3.087182378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwvm" (UniqueName: "kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm") pod "network-check-target-dcdqx" (UID: "2397ea2a-dfea-45e4-8858-e51400496bff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:23.834809 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.834731    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jf2\" (UniqueName: \"kubernetes.io/projected/e915550d-e2e5-4bdb-b20c-6e2e2cf9756d-kube-api-access-48jf2\") pod \"ovnkube-node-6fxqg\" (UID: \"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:23.834809 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.834735    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8ls\" (UniqueName: \"kubernetes.io/projected/d14d47c1-872a-4915-9cc5-f57cf19c52bb-kube-api-access-kb8ls\") pod \"node-ca-z7trl\" (UID: \"d14d47c1-872a-4915-9cc5-f57cf19c52bb\") " pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:23.836417 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.836371    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcrn7\" (UniqueName: \"kubernetes.io/projected/4946c3b1-5fd0-4cb9-80b5-48a624edd688-kube-api-access-jcrn7\") pod \"iptables-alerter-7kjd5\" (UID: \"4946c3b1-5fd0-4cb9-80b5-48a624edd688\") " pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:23.836651 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.836627    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkjd\" (UniqueName: \"kubernetes.io/projected/eba74a0a-d5a5-4174-8a96-4e7ccdba92d2-kube-api-access-kqkjd\") pod \"tuned-rk6gc\" (UID: \"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2\") " pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:23.926779 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926739    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-os-release\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926791    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-socket-dir-parent\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926829    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-hostroot\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926860    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q72vh\" (UniqueName: \"kubernetes.io/projected/d863a6d6-a9ed-4c86-b88f-2541282968b3-kube-api-access-q72vh\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926887    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-os-release\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926900    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-socket-dir-parent\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926895    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926932    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-hostroot\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.926948 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926950    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.926987    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-conf-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927016    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-42swz\" (UniqueName: \"kubernetes.io/projected/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-kube-api-access-42swz\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927043    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cnibin\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927069    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-conf-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927072    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqb7\" (UniqueName: \"kubernetes.io/projected/e5d18530-e2b7-49bf-906a-2969cfb88e1d-kube-api-access-7sqb7\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927108    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cnibin\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927114    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxm6s\" (UniqueName: \"kubernetes.io/projected/7ad17068-4f7d-491f-8d77-61c903d241cb-kube-api-access-pxm6s\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927144    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-os-release\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927165    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9442f5d3-3e19-44ec-8db5-1706c9b11ba2-agent-certs\") pod \"konnectivity-agent-6mk6l\" (UID: \"9442f5d3-3e19-44ec-8db5-1706c9b11ba2\") " pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927189    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-socket-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927212    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-registration-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927235    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927252    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-os-release\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927261    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927294    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ad17068-4f7d-491f-8d77-61c903d241cb-hosts-file\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927321    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-daemon-config\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.927376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927353    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-registration-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927366    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-system-cni-dir\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927410    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-system-cni-dir\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927434    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-system-cni-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927461    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-cni-multus\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927477    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-kubelet\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927504    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-multus-certs\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927521    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ad17068-4f7d-491f-8d77-61c903d241cb-hosts-file\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927531    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad17068-4f7d-491f-8d77-61c903d241cb-tmp-dir\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927559    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927582    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-etc-kubernetes\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927613    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-system-cni-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927614    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9442f5d3-3e19-44ec-8db5-1706c9b11ba2-konnectivity-ca\") pod \"konnectivity-agent-6mk6l\" (UID: \"9442f5d3-3e19-44ec-8db5-1706c9b11ba2\") " pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927620    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-cni-multus\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927646    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-sys-fs\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927650    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-multus-certs\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927656    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-socket-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927671    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-etc-kubernetes\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.928168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927673    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927697    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-kubelet\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927715    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-cni-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927740    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-cnibin\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927766    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-cni-binary-copy\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927793    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-device-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927820    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927848    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-netns\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927877    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-cni-bin\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927893    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-cnibin\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927903    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bl7dc\" (UniqueName: \"kubernetes.io/projected/61545c61-84b8-432a-9443-e9fd8b59060f-kube-api-access-bl7dc\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927948    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad17068-4f7d-491f-8d77-61c903d241cb-tmp-dir\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927953    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.927991    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928014    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-cni-dir\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928019    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-k8s-cni-cncf-io\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928138    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5d18530-e2b7-49bf-906a-2969cfb88e1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928145    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-k8s-cni-cncf-io\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928146    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-multus-daemon-config\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928191    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-sys-fs\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928214    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9442f5d3-3e19-44ec-8db5-1706c9b11ba2-konnectivity-ca\") pod \"konnectivity-agent-6mk6l\" (UID: \"9442f5d3-3e19-44ec-8db5-1706c9b11ba2\") " pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.928223    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928254    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928270    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d863a6d6-a9ed-4c86-b88f-2541282968b3-device-dir\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:23.928305    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:17:24.428287046 +0000 UTC m=+3.183263837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928317    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-run-netns\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928350    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-host-var-lib-cni-bin\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928583    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-cni-binary-copy\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928646    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.929932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.928707    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5d18530-e2b7-49bf-906a-2969cfb88e1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.930337 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.930249    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9442f5d3-3e19-44ec-8db5-1706c9b11ba2-agent-certs\") pod \"konnectivity-agent-6mk6l\" (UID: \"9442f5d3-3e19-44ec-8db5-1706c9b11ba2\") " pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:23.935086 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.935012    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxm6s\" (UniqueName: \"kubernetes.io/projected/7ad17068-4f7d-491f-8d77-61c903d241cb-kube-api-access-pxm6s\") pod \"node-resolver-jqzls\" (UID: \"7ad17068-4f7d-491f-8d77-61c903d241cb\") " pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:23.936681 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.936660    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72vh\" (UniqueName: \"kubernetes.io/projected/d863a6d6-a9ed-4c86-b88f-2541282968b3-kube-api-access-q72vh\") pod \"aws-ebs-csi-driver-node-v8hpn\" (UID: \"d863a6d6-a9ed-4c86-b88f-2541282968b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:23.937424 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.937397    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl7dc\" (UniqueName: \"kubernetes.io/projected/61545c61-84b8-432a-9443-e9fd8b59060f-kube-api-access-bl7dc\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:23.937584 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.937560    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqb7\" (UniqueName: \"kubernetes.io/projected/e5d18530-e2b7-49bf-906a-2969cfb88e1d-kube-api-access-7sqb7\") pod \"multus-additional-cni-plugins-5f87w\" (UID: \"e5d18530-e2b7-49bf-906a-2969cfb88e1d\") " pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:23.937810 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:23.937795    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42swz\" (UniqueName: \"kubernetes.io/projected/c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb-kube-api-access-42swz\") pod \"multus-mm888\" (UID: \"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb\") " pod="openshift-multus/multus-mm888"
May 25 11:17:24.008925 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.008884    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rk6gc"
May 25 11:17:24.016730 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.016704    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:24.029284 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.029254    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7kjd5"
May 25 11:17:24.034039 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.034014    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7trl"
May 25 11:17:24.040672 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.040647    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn"
May 25 11:17:24.047276 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.047255    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jqzls"
May 25 11:17:24.052829 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.052801    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5f87w"
May 25 11:17:24.059436 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.059416    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mm888"
May 25 11:17:24.065037 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.065019    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:24.410599 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.410572    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd863a6d6_a9ed_4c86_b88f_2541282968b3.slice/crio-75286e5a4a91417649df0ca705e640ef7e2f1b84329e742f0e6fee830fc0a0a3 WatchSource:0}: Error finding container 75286e5a4a91417649df0ca705e640ef7e2f1b84329e742f0e6fee830fc0a0a3: Status 404 returned error can't find the container with id 75286e5a4a91417649df0ca705e640ef7e2f1b84329e742f0e6fee830fc0a0a3
May 25 11:17:24.411468 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.411440    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14d47c1_872a_4915_9cc5_f57cf19c52bb.slice/crio-88a3358f8c9489651ce3c7acae1bddfed91f7f7d7d27f46f1745cc0909aa9481 WatchSource:0}: Error finding container 88a3358f8c9489651ce3c7acae1bddfed91f7f7d7d27f46f1745cc0909aa9481: Status 404 returned error can't find the container with id 88a3358f8c9489651ce3c7acae1bddfed91f7f7d7d27f46f1745cc0909aa9481
May 25 11:17:24.412159 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.412137    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode915550d_e2e5_4bdb_b20c_6e2e2cf9756d.slice/crio-5011f5eb78a83acc18cf50ea58ac3d61b17145d09ddcd0399542ceea49cb69d7 WatchSource:0}: Error finding container 5011f5eb78a83acc18cf50ea58ac3d61b17145d09ddcd0399542ceea49cb69d7: Status 404 returned error can't find the container with id 5011f5eb78a83acc18cf50ea58ac3d61b17145d09ddcd0399542ceea49cb69d7
May 25 11:17:24.413469 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.413442    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dc4e95_98e6_4e49_95d6_d3ccedf3cdcb.slice/crio-477561e640d576abbeb191a0ab0a66de3fb3baefd3b71a91e5b309833e159b61 WatchSource:0}: Error finding container 477561e640d576abbeb191a0ab0a66de3fb3baefd3b71a91e5b309833e159b61: Status 404 returned error can't find the container with id 477561e640d576abbeb191a0ab0a66de3fb3baefd3b71a91e5b309833e159b61
May 25 11:17:24.415978 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.415955    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad17068_4f7d_491f_8d77_61c903d241cb.slice/crio-18116c68fcfe53434a748b5ccb383a7e02e8b151c44bb3a7d2a6d6cf493949b6 WatchSource:0}: Error finding container 18116c68fcfe53434a748b5ccb383a7e02e8b151c44bb3a7d2a6d6cf493949b6: Status 404 returned error can't find the container with id 18116c68fcfe53434a748b5ccb383a7e02e8b151c44bb3a7d2a6d6cf493949b6
May 25 11:17:24.417142 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.417108    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4946c3b1_5fd0_4cb9_80b5_48a624edd688.slice/crio-4d955839cd0ad636e094ab3bb1b3644f30451a3ea83336176fad9f2ee6fe5ad0 WatchSource:0}: Error finding container 4d955839cd0ad636e094ab3bb1b3644f30451a3ea83336176fad9f2ee6fe5ad0: Status 404 returned error can't find the container with id 4d955839cd0ad636e094ab3bb1b3644f30451a3ea83336176fad9f2ee6fe5ad0
May 25 11:17:24.417906 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.417884    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d18530_e2b7_49bf_906a_2969cfb88e1d.slice/crio-3cfec712d6afae08bd013f797c7be2322fe85a34bdf526e157fdc32209088f5a WatchSource:0}: Error finding container 3cfec712d6afae08bd013f797c7be2322fe85a34bdf526e157fdc32209088f5a: Status 404 returned error can't find the container with id 3cfec712d6afae08bd013f797c7be2322fe85a34bdf526e157fdc32209088f5a
May 25 11:17:24.419862 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.418582    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba74a0a_d5a5_4174_8a96_4e7ccdba92d2.slice/crio-57803067bb1d790380bceaeddc7cfa2873cd5e542efcd676082f93a3b106a86e WatchSource:0}: Error finding container 57803067bb1d790380bceaeddc7cfa2873cd5e542efcd676082f93a3b106a86e: Status 404 returned error can't find the container with id 57803067bb1d790380bceaeddc7cfa2873cd5e542efcd676082f93a3b106a86e
May 25 11:17:24.419862 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:17:24.419610    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9442f5d3_3e19_44ec_8db5_1706c9b11ba2.slice/crio-71e359262d5a5f2e9865364634ae3df42eaf0da844fe148271ec763d748583eb WatchSource:0}: Error finding container 71e359262d5a5f2e9865364634ae3df42eaf0da844fe148271ec763d748583eb: Status 404 returned error can't find the container with id 71e359262d5a5f2e9865364634ae3df42eaf0da844fe148271ec763d748583eb
May 25 11:17:24.431055 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.430789    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:24.431055 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.430863    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:24.431055 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:24.430976    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:24.431055 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:24.430999    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:24.431055 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:24.431017    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:24.431055 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:24.431029    2566 projected.go:194] Error preparing data for projected volume kube-api-access-zcwvm for pod openshift-network-diagnostics/network-check-target-dcdqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:24.431055 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:24.431030    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:17:25.431012814 +0000 UTC m=+4.185989602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:24.431562 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:24.431094    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm podName:2397ea2a-dfea-45e4-8858-e51400496bff nodeName:}" failed. No retries permitted until 2026-05-25 11:17:25.431078791 +0000 UTC m=+4.186055574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwvm" (UniqueName: "kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm") pod "network-check-target-dcdqx" (UID: "2397ea2a-dfea-45e4-8858-e51400496bff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:24.758072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.757844    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:22 +0000 UTC" deadline="2028-02-04 11:18:21.39059053 +0000 UTC"
May 25 11:17:24.758072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.758074    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14880h0m56.632521855s"
May 25 11:17:24.831239 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.830706    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:24.831239 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:24.830830    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:24.844672 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.844006    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal" event={"ID":"324b37d0baef0cf5218e4c8fcb924549","Type":"ContainerStarted","Data":"b368aa5994ef2d9acfdc19406ab99d506e51c696c1b50722912ba856d3b271b6"}
May 25 11:17:24.847971 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.846692    2566 generic.go:358] "Generic (PLEG): container finished" podID="618d39cc139f57474525689dc86df07c" containerID="b83f439753266b258ae945b029b46b90976824f2a0755ee8f8a9b1431aef7e1d" exitCode=0
May 25 11:17:24.847971 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.846764    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" event={"ID":"618d39cc139f57474525689dc86df07c","Type":"ContainerDied","Data":"b83f439753266b258ae945b029b46b90976824f2a0755ee8f8a9b1431aef7e1d"}
May 25 11:17:24.852645 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.852618    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6mk6l" event={"ID":"9442f5d3-3e19-44ec-8db5-1706c9b11ba2","Type":"ContainerStarted","Data":"71e359262d5a5f2e9865364634ae3df42eaf0da844fe148271ec763d748583eb"}
May 25 11:17:24.857592 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.856462    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal" podStartSLOduration=1.8564475649999999 podStartE2EDuration="1.856447565s" podCreationTimestamp="2026-05-25 11:17:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:24.856429699 +0000 UTC m=+3.611406494" watchObservedRunningTime="2026-05-25 11:17:24.856447565 +0000 UTC m=+3.611424353"
May 25 11:17:24.857961 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.857937    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerStarted","Data":"3cfec712d6afae08bd013f797c7be2322fe85a34bdf526e157fdc32209088f5a"}
May 25 11:17:24.867602 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.867577    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rk6gc" event={"ID":"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2","Type":"ContainerStarted","Data":"57803067bb1d790380bceaeddc7cfa2873cd5e542efcd676082f93a3b106a86e"}
May 25 11:17:24.878354 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.878305    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7trl" event={"ID":"d14d47c1-872a-4915-9cc5-f57cf19c52bb","Type":"ContainerStarted","Data":"88a3358f8c9489651ce3c7acae1bddfed91f7f7d7d27f46f1745cc0909aa9481"}
May 25 11:17:24.882388 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.882210    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn" event={"ID":"d863a6d6-a9ed-4c86-b88f-2541282968b3","Type":"ContainerStarted","Data":"75286e5a4a91417649df0ca705e640ef7e2f1b84329e742f0e6fee830fc0a0a3"}
May 25 11:17:24.884170 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.884122    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7kjd5" event={"ID":"4946c3b1-5fd0-4cb9-80b5-48a624edd688","Type":"ContainerStarted","Data":"4d955839cd0ad636e094ab3bb1b3644f30451a3ea83336176fad9f2ee6fe5ad0"}
May 25 11:17:24.893726 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.893700    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jqzls" event={"ID":"7ad17068-4f7d-491f-8d77-61c903d241cb","Type":"ContainerStarted","Data":"18116c68fcfe53434a748b5ccb383a7e02e8b151c44bb3a7d2a6d6cf493949b6"}
May 25 11:17:24.901650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.901625    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mm888" event={"ID":"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb","Type":"ContainerStarted","Data":"477561e640d576abbeb191a0ab0a66de3fb3baefd3b71a91e5b309833e159b61"}
May 25 11:17:24.904956 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:24.904921    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"5011f5eb78a83acc18cf50ea58ac3d61b17145d09ddcd0399542ceea49cb69d7"}
May 25 11:17:25.440332 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:25.440289    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:25.440523 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:25.440371    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:25.440523 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:25.440492    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:25.440673 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:25.440571    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:17:27.440551622 +0000 UTC m=+6.195528412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:25.441008 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:25.440982    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:25.441008 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:25.441012    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:25.441161 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:25.441024    2566 projected.go:194] Error preparing data for projected volume kube-api-access-zcwvm for pod openshift-network-diagnostics/network-check-target-dcdqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:25.441161 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:25.441072    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm podName:2397ea2a-dfea-45e4-8858-e51400496bff nodeName:}" failed. No retries permitted until 2026-05-25 11:17:27.441055216 +0000 UTC m=+6.196031991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwvm" (UniqueName: "kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm") pod "network-check-target-dcdqx" (UID: "2397ea2a-dfea-45e4-8858-e51400496bff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:25.831774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:25.831682    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:25.832220 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:25.831835    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:25.940070 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:25.940034    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" event={"ID":"618d39cc139f57474525689dc86df07c","Type":"ContainerStarted","Data":"75b23d819efbbd6746eaadab47bf90aa95206c64f4154f28ccd7994dfe7dd11c"}
May 25 11:17:26.832065 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:26.831557    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:26.832065 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:26.831701    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:27.458454 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:27.458414    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:27.458643 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:27.458523    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:27.461190 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:27.460785    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:27.461190 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:27.460817    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:27.461190 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:27.460842    2566 projected.go:194] Error preparing data for projected volume kube-api-access-zcwvm for pod openshift-network-diagnostics/network-check-target-dcdqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:27.461190 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:27.460894    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:27.461190 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:27.460930    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm podName:2397ea2a-dfea-45e4-8858-e51400496bff nodeName:}" failed. No retries permitted until 2026-05-25 11:17:31.460889723 +0000 UTC m=+10.215866514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwvm" (UniqueName: "kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm") pod "network-check-target-dcdqx" (UID: "2397ea2a-dfea-45e4-8858-e51400496bff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:27.461190 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:27.460981    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:17:31.4609619 +0000 UTC m=+10.215938679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:27.834193 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:27.834117    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:27.834627 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:27.834265    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:28.831352 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:28.830890    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:28.831352 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:28.831009    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:29.832527 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:29.832121    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:29.832527 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:29.832240    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:30.831591 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:30.831340    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:30.831591 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:30.831480    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:31.493704 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:31.493667    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:31.494159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:31.493739    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:31.494159 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:31.493817    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:31.494159 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:31.493875    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:31.494159 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:31.493890    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:31.494159 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:31.493901    2566 projected.go:194] Error preparing data for projected volume kube-api-access-zcwvm for pod openshift-network-diagnostics/network-check-target-dcdqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:31.494159 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:31.493907    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:17:39.493885033 +0000 UTC m=+18.248861824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:31.494159 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:31.493946    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm podName:2397ea2a-dfea-45e4-8858-e51400496bff nodeName:}" failed. No retries permitted until 2026-05-25 11:17:39.49393346 +0000 UTC m=+18.248910237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwvm" (UniqueName: "kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm") pod "network-check-target-dcdqx" (UID: "2397ea2a-dfea-45e4-8858-e51400496bff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:31.832304 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:31.831752    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:31.832304 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:31.831880    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:32.831438 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:32.831394    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:32.832051 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:32.831578    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:33.831368 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:33.831333    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:33.831521 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:33.831450    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:34.830837 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:34.830804    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:34.831299 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:34.830932    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:35.384812 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.384745    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" podStartSLOduration=12.384730855 podStartE2EDuration="12.384730855s" podCreationTimestamp="2026-05-25 11:17:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:25.954267589 +0000 UTC m=+4.709244386" watchObservedRunningTime="2026-05-25 11:17:35.384730855 +0000 UTC m=+14.139707651"
May 25 11:17:35.385316 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.385296    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8smhx"]
May 25 11:17:35.397001 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.396973    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.397130 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:35.397052    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:35.524191 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.524150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6275e60c-94fe-4622-a186-d02cea5494d0-kubelet-config\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.524191 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.524193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6275e60c-94fe-4622-a186-d02cea5494d0-dbus\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.524446 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.524290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.625514 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.625477    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.625751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.625561    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6275e60c-94fe-4622-a186-d02cea5494d0-kubelet-config\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.625751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.625582    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6275e60c-94fe-4622-a186-d02cea5494d0-dbus\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.625751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.625659    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6275e60c-94fe-4622-a186-d02cea5494d0-kubelet-config\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.625751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.625740    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6275e60c-94fe-4622-a186-d02cea5494d0-dbus\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:35.625963 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:35.625768    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:35.625963 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:35.625839    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret podName:6275e60c-94fe-4622-a186-d02cea5494d0 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:36.125819215 +0000 UTC m=+14.880796009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret") pod "global-pull-secret-syncer-8smhx" (UID: "6275e60c-94fe-4622-a186-d02cea5494d0") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:35.831527 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:35.831445    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:35.831985 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:35.831595    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:36.129318 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:36.129287    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:36.129507 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:36.129452    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:36.129597 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:36.129526    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret podName:6275e60c-94fe-4622-a186-d02cea5494d0 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:37.129506581 +0000 UTC m=+15.884483355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret") pod "global-pull-secret-syncer-8smhx" (UID: "6275e60c-94fe-4622-a186-d02cea5494d0") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:36.831503 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:36.831461    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:36.831669 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:36.831611    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:37.136654 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:37.136616    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:37.136833 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:37.136776    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:37.136901 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:37.136844    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret podName:6275e60c-94fe-4622-a186-d02cea5494d0 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:39.136829013 +0000 UTC m=+17.891805798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret") pod "global-pull-secret-syncer-8smhx" (UID: "6275e60c-94fe-4622-a186-d02cea5494d0") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:37.830708 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:37.830661    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:37.830877 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:37.830746    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:37.830877 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:37.830851    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:37.831006 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:37.830983    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:38.831465 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:38.831423    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:38.831936 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:38.831571    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:39.151614 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:39.151573    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:39.151799 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.151747    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:39.151861 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.151831    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret podName:6275e60c-94fe-4622-a186-d02cea5494d0 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:43.151810792 +0000 UTC m=+21.906787582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret") pod "global-pull-secret-syncer-8smhx" (UID: "6275e60c-94fe-4622-a186-d02cea5494d0") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:39.554127 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:39.554036    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:39.554127 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:39.554108    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:39.554358 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.554245    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:39.554358 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.554261    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:39.554358 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.554286    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:39.554358 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.554296    2566 projected.go:194] Error preparing data for projected volume kube-api-access-zcwvm for pod openshift-network-diagnostics/network-check-target-dcdqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:39.554358 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.554322    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:17:55.55430606 +0000 UTC m=+34.309282839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:39.554358 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.554361    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm podName:2397ea2a-dfea-45e4-8858-e51400496bff nodeName:}" failed. No retries permitted until 2026-05-25 11:17:55.554333166 +0000 UTC m=+34.309309941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwvm" (UniqueName: "kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm") pod "network-check-target-dcdqx" (UID: "2397ea2a-dfea-45e4-8858-e51400496bff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:39.831059 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:39.830965    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:39.831212 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:39.830968    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:39.831212 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.831105    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:39.831212 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:39.831196    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:40.830848 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:40.830816    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:40.831282 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:40.830926    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:41.832239 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:41.832204    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:41.832719 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:41.832321    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:41.832719 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:41.832374    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:41.832719 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:41.832477    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:42.831812 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.831461    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:42.831812 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:42.831605    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:42.975834 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.975664    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"70e311d33bffea0fabaabf080f8136ad0c4bb799002134d013e3728ac285ec23"}
May 25 11:17:42.976917 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.976894    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6mk6l" event={"ID":"9442f5d3-3e19-44ec-8db5-1706c9b11ba2","Type":"ContainerStarted","Data":"3c32b5674dbfc43428ad8da5202d451ff75087779cb91ff5490b814c2fca33a3"}
May 25 11:17:42.978170 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.978144    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerStarted","Data":"21405394a6e81b29e3547dc3c1dd02d307d9b6b9d78bc7ac5ec035a9d8248a8b"}
May 25 11:17:42.979253 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.979230    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rk6gc" event={"ID":"eba74a0a-d5a5-4174-8a96-4e7ccdba92d2","Type":"ContainerStarted","Data":"e82b6326525404b987688f6ac1d2b573c054716154d191d5921c7ea27946bf0e"}
May 25 11:17:42.980487 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.980462    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7trl" event={"ID":"d14d47c1-872a-4915-9cc5-f57cf19c52bb","Type":"ContainerStarted","Data":"d3e76ab524de77697cd7b03b4e53c4fff2ff74e589af6b1708220e3cc395c323"}
May 25 11:17:42.981863 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.981843    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn" event={"ID":"d863a6d6-a9ed-4c86-b88f-2541282968b3","Type":"ContainerStarted","Data":"320c2d8df041d3cfc508b5cec638d54197114c8e0d966b1eda628312768d557e"}
May 25 11:17:42.983087 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.983007    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jqzls" event={"ID":"7ad17068-4f7d-491f-8d77-61c903d241cb","Type":"ContainerStarted","Data":"5dea4463678c7b13b84ca90353c0c6c2f8bf7f0255c6efdea16e99b358dc73d3"}
May 25 11:17:42.984785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.984653    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mm888" event={"ID":"c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb","Type":"ContainerStarted","Data":"df31ec903ec9c5b9990cc653472560277893c4f65ae718dbe9e7ad8f0561e2c3"}
May 25 11:17:42.990250 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:42.990197    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6mk6l" podStartSLOduration=2.775645187 podStartE2EDuration="20.990179655s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.422910286 +0000 UTC m=+3.177887074" lastFinishedPulling="2026-05-25 11:17:42.637444754 +0000 UTC m=+21.392421542" observedRunningTime="2026-05-25 11:17:42.989879327 +0000 UTC m=+21.744856124" watchObservedRunningTime="2026-05-25 11:17:42.990179655 +0000 UTC m=+21.745156452"
May 25 11:17:43.018785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.018248    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:43.019387 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.019366    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:43.038237 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.038193    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rk6gc" podStartSLOduration=3.758545013 podStartE2EDuration="22.038173609s" podCreationTimestamp="2026-05-25 11:17:21 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.422960669 +0000 UTC m=+3.177937447" lastFinishedPulling="2026-05-25 11:17:42.702589268 +0000 UTC m=+21.457566043" observedRunningTime="2026-05-25 11:17:43.02378424 +0000 UTC m=+21.778761037" watchObservedRunningTime="2026-05-25 11:17:43.038173609 +0000 UTC m=+21.793150408"
May 25 11:17:43.038942 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.038578    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-jqzls" podStartSLOduration=2.776226074 podStartE2EDuration="21.038568377s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.418007672 +0000 UTC m=+3.172984447" lastFinishedPulling="2026-05-25 11:17:42.680349969 +0000 UTC m=+21.435326750" observedRunningTime="2026-05-25 11:17:43.037804724 +0000 UTC m=+21.792781521" watchObservedRunningTime="2026-05-25 11:17:43.038568377 +0000 UTC m=+21.793545172"
May 25 11:17:43.055966 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.055909    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-mm888" podStartSLOduration=2.575072526 podStartE2EDuration="21.055886711s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.415520363 +0000 UTC m=+3.170497136" lastFinishedPulling="2026-05-25 11:17:42.896334532 +0000 UTC m=+21.651311321" observedRunningTime="2026-05-25 11:17:43.054247901 +0000 UTC m=+21.809224696" watchObservedRunningTime="2026-05-25 11:17:43.055886711 +0000 UTC m=+21.810863508"
May 25 11:17:43.079136 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.079067    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z7trl" podStartSLOduration=2.790864349 podStartE2EDuration="21.079036639s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.414421134 +0000 UTC m=+3.169397911" lastFinishedPulling="2026-05-25 11:17:42.702593414 +0000 UTC m=+21.457570201" observedRunningTime="2026-05-25 11:17:43.078980634 +0000 UTC m=+21.833957430" watchObservedRunningTime="2026-05-25 11:17:43.079036639 +0000 UTC m=+21.834013438"
May 25 11:17:43.181314 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.181266    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:43.181437 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:43.181402    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:43.181521 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:43.181468    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret podName:6275e60c-94fe-4622-a186-d02cea5494d0 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:51.181451764 +0000 UTC m=+29.936428548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret") pod "global-pull-secret-syncer-8smhx" (UID: "6275e60c-94fe-4622-a186-d02cea5494d0") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:43.831322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.831241    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:43.831515 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.831245    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:43.831515 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:43.831353    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:43.831515 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:43.831443    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:43.987328 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.987294    2566 generic.go:358] "Generic (PLEG): container finished" podID="e5d18530-e2b7-49bf-906a-2969cfb88e1d" containerID="21405394a6e81b29e3547dc3c1dd02d307d9b6b9d78bc7ac5ec035a9d8248a8b" exitCode=0
May 25 11:17:43.988025 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.987381    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerDied","Data":"21405394a6e81b29e3547dc3c1dd02d307d9b6b9d78bc7ac5ec035a9d8248a8b"}
May 25 11:17:43.989884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.989843    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"45b0f917e44c56a5480d0bac8b83c2ea48274aa315b9cd74229fc7bcc1c462e9"}
May 25 11:17:43.989884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.989885    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"fb3948a46a95afa8a362c1592a6ced403c82d2f0a7e23590360d475635a77178"}
May 25 11:17:43.990057 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.989901    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"16e0e8772819d5732c00fb66c23781caf81be66f655c8892584f6799fbfb48c4"}
May 25 11:17:43.990057 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.989913    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"422f25f88d3342ad6715fc11578310cd5ca2ed1ebf348cb334e34c66c8a22a8b"}
May 25 11:17:43.990057 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:43.989923    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"f6379a4db6c75007af513c8905fbe63432eb812ec698f678712a085a96931fba"}
May 25 11:17:44.195495 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.195472    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 11:17:44.780603 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.780476    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:17:44.19548924Z","UUID":"4b77389a-9308-43c0-8ee4-e9cb6aa7b9b1","Handler":null,"Name":"","Endpoint":""}
May 25 11:17:44.783823 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.783802    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 11:17:44.783955 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.783832    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 11:17:44.831099 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.831064    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:44.831273 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:44.831191    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:44.994069 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.994028    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn" event={"ID":"d863a6d6-a9ed-4c86-b88f-2541282968b3","Type":"ContainerStarted","Data":"530889c6cf80f085ebea87fb9b61e37a0cb59ade14768dbc43d7224f0819dce3"}
May 25 11:17:44.995507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.995478    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7kjd5" event={"ID":"4946c3b1-5fd0-4cb9-80b5-48a624edd688","Type":"ContainerStarted","Data":"7d40a655474f5ffed7165ba2ae0717240db0e4282d3dc2d8b0777a76ff258448"}
May 25 11:17:44.995507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:44.995508    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 11:17:45.831505 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:45.831238    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:45.831695 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:45.831250    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:45.831695 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:45.831592    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:45.831695 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:45.831669    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:45.999724 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:45.999686    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn" event={"ID":"d863a6d6-a9ed-4c86-b88f-2541282968b3","Type":"ContainerStarted","Data":"915583bd99e229261aa976ba5c2f90b16c2f8c82bba46fd20445dc7e8e56ca18"}
May 25 11:17:46.003055 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:46.003018    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"c4095d2ca53bce4e1d10d90328f7b764bb5830e2c424fb44922ece4a53036d54"}
May 25 11:17:46.016770 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:46.016727    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8hpn" podStartSLOduration=3.396898397 podStartE2EDuration="24.016711843s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.412714106 +0000 UTC m=+3.167690895" lastFinishedPulling="2026-05-25 11:17:45.03252756 +0000 UTC m=+23.787504341" observedRunningTime="2026-05-25 11:17:46.016439548 +0000 UTC m=+24.771416345" watchObservedRunningTime="2026-05-25 11:17:46.016711843 +0000 UTC m=+24.771688639"
May 25 11:17:46.017075 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:46.017053    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7kjd5" podStartSLOduration=6.727778966 podStartE2EDuration="25.017047586s" podCreationTimestamp="2026-05-25 11:17:21 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.418977217 +0000 UTC m=+3.173954006" lastFinishedPulling="2026-05-25 11:17:42.70824585 +0000 UTC m=+21.463222626" observedRunningTime="2026-05-25 11:17:45.010681654 +0000 UTC m=+23.765658463" watchObservedRunningTime="2026-05-25 11:17:46.017047586 +0000 UTC m=+24.772024382"
May 25 11:17:46.831376 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:46.831334    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:46.831617 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:46.831472    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:47.831356 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:47.831266    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:47.831356 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:47.831289    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:47.831960 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:47.831399    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:47.831960 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:47.831523    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:48.830974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:48.830742    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:48.831117 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:48.830994    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:49.010358 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:49.010321    2566 generic.go:358] "Generic (PLEG): container finished" podID="e5d18530-e2b7-49bf-906a-2969cfb88e1d" containerID="600be46bcf17850c5243c285b0fd56f1526203c61e40da425352326a86cd2a53" exitCode=0
May 25 11:17:49.011141 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:49.010407    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerDied","Data":"600be46bcf17850c5243c285b0fd56f1526203c61e40da425352326a86cd2a53"}
May 25 11:17:49.017275 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:49.017243    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" event={"ID":"e915550d-e2e5-4bdb-b20c-6e2e2cf9756d","Type":"ContainerStarted","Data":"ebb6aae77be08d6a1fd776af6426f4dd105befbb150db95f7170509d7346aff9"}
May 25 11:17:49.017585 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:49.017565    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:49.033028 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:49.033003    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:49.831649 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:49.831609    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:49.831853 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:49.831616    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:49.831853 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:49.831717    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:49.831853 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:49.831836    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:50.006451 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.006394    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg" podStartSLOduration=10.699429055 podStartE2EDuration="29.00637176s" podCreationTimestamp="2026-05-25 11:17:21 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.414865341 +0000 UTC m=+3.169842123" lastFinishedPulling="2026-05-25 11:17:42.721808042 +0000 UTC m=+21.476784828" observedRunningTime="2026-05-25 11:17:49.063142786 +0000 UTC m=+27.818119583" watchObservedRunningTime="2026-05-25 11:17:50.00637176 +0000 UTC m=+28.761348557"
May 25 11:17:50.007570 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.007526    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8smhx"]
May 25 11:17:50.010165 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.010122    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dcdqx"]
May 25 11:17:50.010275 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.010261    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:50.010445 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:50.010418    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:50.018229 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.018197    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xm9qk"]
May 25 11:17:50.018867 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.018841    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:50.018974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.018845    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:50.019023 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:50.018979    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:50.019070 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:50.019043    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:50.019120 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.019068    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:50.019120 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.018945    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 11:17:50.039875 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:50.039847    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:51.022590 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:51.022555    2566 generic.go:358] "Generic (PLEG): container finished" podID="e5d18530-e2b7-49bf-906a-2969cfb88e1d" containerID="aeb374a55aa478966743450013151e57d24c9ae5be5ad7f13495c52cb9214cd1" exitCode=0
May 25 11:17:51.022939 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:51.022636    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerDied","Data":"aeb374a55aa478966743450013151e57d24c9ae5be5ad7f13495c52cb9214cd1"}
May 25 11:17:51.022939 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:51.022880    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 11:17:51.244673 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:51.244638    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:51.244854 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:51.244789    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:51.244854 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:51.244850    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret podName:6275e60c-94fe-4622-a186-d02cea5494d0 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:07.244833585 +0000 UTC m=+45.999810373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret") pod "global-pull-secret-syncer-8smhx" (UID: "6275e60c-94fe-4622-a186-d02cea5494d0") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:51.831991 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:51.831958    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:51.832131 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:51.832051    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:51.832190 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:51.832135    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:51.832235 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:51.832222    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:51.832287 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:51.832261    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:51.832320 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:51.832292    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:52.026681 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:52.026455    2566 generic.go:358] "Generic (PLEG): container finished" podID="e5d18530-e2b7-49bf-906a-2969cfb88e1d" containerID="2fdd59e9b0855719797c30dc4f00ba9513204dddf35675d37c46ffb4d58570bc" exitCode=0
May 25 11:17:52.027107 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:52.026529    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerDied","Data":"2fdd59e9b0855719797c30dc4f00ba9513204dddf35675d37c46ffb4d58570bc"}
May 25 11:17:52.027107 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:52.026833    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 11:17:52.877859 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:52.877822    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:52.878035 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:52.877987    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 11:17:52.878509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:52.878485    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6mk6l"
May 25 11:17:53.831483 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:53.831448    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:53.832144 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:53.831448    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:53.832144 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:53.831591    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dcdqx" podUID="2397ea2a-dfea-45e4-8858-e51400496bff"
May 25 11:17:53.832144 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:53.831454    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:53.832144 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:53.831678    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8smhx" podUID="6275e60c-94fe-4622-a186-d02cea5494d0"
May 25 11:17:53.832144 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:53.831749    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:17:53.882751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:53.882517    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:53.882982 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:53.882804    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 11:17:53.894470 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:53.894435    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6fxqg"
May 25 11:17:55.523558 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.523488    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeReady"
May 25 11:17:55.524128 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.523644    2566 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 11:17:55.575653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.575532    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:55.575653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.575638    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:55.575867 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.575664    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:55.575867 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.575731    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.575709221 +0000 UTC m=+66.330685998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:55.575867 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.575785    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:55.575867 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.575802    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:55.575867 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.575815    2566 projected.go:194] Error preparing data for projected volume kube-api-access-zcwvm for pod openshift-network-diagnostics/network-check-target-dcdqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:55.576072 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.575870    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm podName:2397ea2a-dfea-45e4-8858-e51400496bff nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.575855584 +0000 UTC m=+66.330832360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwvm" (UniqueName: "kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm") pod "network-check-target-dcdqx" (UID: "2397ea2a-dfea-45e4-8858-e51400496bff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:55.582832 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.582290    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-779dd5d66c-dw82q"]
May 25 11:17:55.603247 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.603220    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d7gl7"]
May 25 11:17:55.603424 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.603398    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.607469 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.607444    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 11:17:55.607688 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.607444    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-srlsx\""
May 25 11:17:55.607688 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.607449    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 11:17:55.607688 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.607667    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 11:17:55.613465 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.612796    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 11:17:55.617044 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.617017    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tnpqg"]
May 25 11:17:55.617207 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.617186    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.620785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.620764    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t8nfn\""
May 25 11:17:55.621084 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.621066    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 11:17:55.621435 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.621290    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 11:17:55.637179 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.637150    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-779dd5d66c-dw82q"]
May 25 11:17:55.637179 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.637176    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnpqg"]
May 25 11:17:55.637179 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.637186    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7gl7"]
May 25 11:17:55.637450 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.637319    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:55.640493 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.640472    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 11:17:55.640680 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.640518    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjf4k\""
May 25 11:17:55.640807 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.640791    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 11:17:55.640881 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.640792    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 11:17:55.777028 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.776994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-trusted-ca\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.777028 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvgf4\" (UniqueName: \"kubernetes.io/projected/26513b67-e7e4-43c9-8a09-d496226e067e-kube-api-access-qvgf4\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.777231 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:55.777231 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-image-registry-private-configuration\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.777231 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/629466b0-845b-4707-8e9d-ad1dd1e2df05-ca-trust-extracted\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.777231 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-certificates\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.777231 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26513b67-e7e4-43c9-8a09-d496226e067e-tmp-dir\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.777426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9dd2\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-kube-api-access-n9dd2\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.777426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ww7q\" (UniqueName: \"kubernetes.io/projected/608f8a96-f300-4efe-89d2-51ca889a3e67-kube-api-access-4ww7q\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:55.777426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777280    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.777426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.777426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-installation-pull-secrets\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.777426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26513b67-e7e4-43c9-8a09-d496226e067e-config-volume\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.777426 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.777354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-bound-sa-token\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.831455 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.831373    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:17:55.831455 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.831403    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:17:55.831455 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.831373    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:17:55.834356 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.834308    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:17:55.834513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.834390    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:17:55.834513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.834449    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:17:55.834852 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.834835    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bl7fp\""
May 25 11:17:55.834852 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.834844    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:17:55.834988 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.834928    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j5m5s\""
May 25 11:17:55.877829 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.877791    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n9dd2\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-kube-api-access-n9dd2\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.877843    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4ww7q\" (UniqueName: \"kubernetes.io/projected/608f8a96-f300-4efe-89d2-51ca889a3e67-kube-api-access-4ww7q\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:55.878011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.877874    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.878011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.877912    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.877938    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-installation-pull-secrets\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.877965    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26513b67-e7e4-43c9-8a09-d496226e067e-config-volume\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.878011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.877994    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-bound-sa-token\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878051    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-trusted-ca\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878086    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qvgf4\" (UniqueName: \"kubernetes.io/projected/26513b67-e7e4-43c9-8a09-d496226e067e-kube-api-access-qvgf4\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878113    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878143    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-image-registry-private-configuration\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878170    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/629466b0-845b-4707-8e9d-ad1dd1e2df05-ca-trust-extracted\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878212    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-certificates\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878244    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26513b67-e7e4-43c9-8a09-d496226e067e-tmp-dir\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.878306 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.878269    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:55.878691 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.878344    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.378326698 +0000 UTC m=+35.133303489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:17:55.878691 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878521    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26513b67-e7e4-43c9-8a09-d496226e067e-tmp-dir\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.878691 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878662    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/629466b0-845b-4707-8e9d-ad1dd1e2df05-ca-trust-extracted\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.878817 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878685    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26513b67-e7e4-43c9-8a09-d496226e067e-config-volume\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:55.878859 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.878830    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:55.878925 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.878909    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:55.878978 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.878930    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:17:55.879028 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.878915    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.378894729 +0000 UTC m=+35.133871505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:17:55.879028 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.878982    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-certificates\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.879028 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:55.879003    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.378987383 +0000 UTC m=+35.133964157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:17:55.879135 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.879035    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-trusted-ca\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.882664 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.882640    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-image-registry-private-configuration\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.882792 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.882685    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-installation-pull-secrets\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.887477 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.887426    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-bound-sa-token\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.887622 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.887518    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ww7q\" (UniqueName: \"kubernetes.io/projected/608f8a96-f300-4efe-89d2-51ca889a3e67-kube-api-access-4ww7q\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:55.888682 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.888644    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9dd2\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-kube-api-access-n9dd2\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:55.888860 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:55.888833    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvgf4\" (UniqueName: \"kubernetes.io/projected/26513b67-e7e4-43c9-8a09-d496226e067e-kube-api-access-qvgf4\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:56.382652 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:56.382612    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:56.382850 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:56.382705    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:56.382850 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:56.382732    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:56.382850 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:56.382796    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:56.383016 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:56.382869    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:56.383016 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:56.382886    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:57.382863497 +0000 UTC m=+36.137840274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:17:56.383016 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:56.382895    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:56.383016 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:56.382909    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:17:56.383016 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:56.382943    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:57.382926884 +0000 UTC m=+36.137903660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:17:56.383016 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:56.382963    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:57.382953201 +0000 UTC m=+36.137929981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:17:57.391670 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:57.391606    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:57.391710    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:57.391751    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:57.391759    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:57.391821    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:57.391849    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:59.391828485 +0000 UTC m=+38.146805262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:57.391859    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:57.391875    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:57.391881    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:59.391863106 +0000 UTC m=+38.146839881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:17:57.392346 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:57.391919    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:59.391903948 +0000 UTC m=+38.146880723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:17:59.044371 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:59.044127    2566 generic.go:358] "Generic (PLEG): container finished" podID="e5d18530-e2b7-49bf-906a-2969cfb88e1d" containerID="1f7ceebd4264143e4b595375c906dc15afcd588f07838793ef4d80608f16b3ff" exitCode=0
May 25 11:17:59.044371 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:59.044199    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerDied","Data":"1f7ceebd4264143e4b595375c906dc15afcd588f07838793ef4d80608f16b3ff"}
May 25 11:17:59.409030 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:59.408978    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:17:59.409030 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:59.409030    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:17:59.409094    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:59.409130    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:59.409185    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:59.409195    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:59.409200    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:59.409200    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:03.409182443 +0000 UTC m=+42.164159231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:59.409243    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:03.409231973 +0000 UTC m=+42.164208747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:17:59.409345 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:17:59.409258    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:03.409251883 +0000 UTC m=+42.164228657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:18:00.049105 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:00.049064    2566 generic.go:358] "Generic (PLEG): container finished" podID="e5d18530-e2b7-49bf-906a-2969cfb88e1d" containerID="2a5b765840f6a9454b35eb53dc0e87db614316af33f8523340d2ca826a857ab3" exitCode=0
May 25 11:18:00.049454 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:00.049131    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerDied","Data":"2a5b765840f6a9454b35eb53dc0e87db614316af33f8523340d2ca826a857ab3"}
May 25 11:18:01.054136 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:01.054097    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5f87w" event={"ID":"e5d18530-e2b7-49bf-906a-2969cfb88e1d","Type":"ContainerStarted","Data":"66b2c2825ca2c0a13eb533883a4186d649103f9cf103db662a5ffdc1c0fc6a80"}
May 25 11:18:01.081923 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:01.081844    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5f87w" podStartSLOduration=5.306776809 podStartE2EDuration="39.081822859s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="2026-05-25 11:17:24.420521987 +0000 UTC m=+3.175498762" lastFinishedPulling="2026-05-25 11:17:58.195568038 +0000 UTC m=+36.950544812" observedRunningTime="2026-05-25 11:18:01.080462441 +0000 UTC m=+39.835439240" watchObservedRunningTime="2026-05-25 11:18:01.081822859 +0000 UTC m=+39.836799691"
May 25 11:18:02.398585 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.398535    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"]
May 25 11:18:02.433138 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.433111    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"]
May 25 11:18:02.433320 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.433281    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.436300 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.436270    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 11:18:02.436717 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.436696    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 11:18:02.436834 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.436700    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 25 11:18:02.436834 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.436700    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 11:18:02.447206 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.447184    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"]
May 25 11:18:02.447206 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.447208    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"]
May 25 11:18:02.447343 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.447219    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"]
May 25 11:18:02.447343 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.447308    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.450239 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.450219    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7ghnj\""
May 25 11:18:02.450361 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.450321    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 25 11:18:02.461418 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.461393    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"]
May 25 11:18:02.461418 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.461398    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.466513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.466491    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 25 11:18:02.466651 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.466522    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 25 11:18:02.466651 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.466491    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 25 11:18:02.466855 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.466836    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 25 11:18:02.531646 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.531614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg88n\" (UniqueName: \"kubernetes.io/projected/94860428-39ed-41be-b019-fb867d18c0a7-kube-api-access-mg88n\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.531817 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.531713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94860428-39ed-41be-b019-fb867d18c0a7-tmp\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.531817 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.531749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/94860428-39ed-41be-b019-fb867d18c0a7-klusterlet-config\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.633109 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633072    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mg88n\" (UniqueName: \"kubernetes.io/projected/94860428-39ed-41be-b019-fb867d18c0a7-kube-api-access-mg88n\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.633109 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-ca\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.633323 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.633323 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffkd\" (UniqueName: \"kubernetes.io/projected/68751f3d-01e1-4203-a68f-6282bba51534-kube-api-access-jffkd\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.633323 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-hub\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.633323 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2852d0-4c85-4a25-8805-24cdeeca6f8c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll\" (UID: \"0c2852d0-4c85-4a25-8805-24cdeeca6f8c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.633523 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633363    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94860428-39ed-41be-b019-fb867d18c0a7-tmp\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.633523 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/68751f3d-01e1-4203-a68f-6282bba51534-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.633523 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbn9q\" (UniqueName: \"kubernetes.io/projected/0c2852d0-4c85-4a25-8805-24cdeeca6f8c-kube-api-access-jbn9q\") pod \"managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll\" (UID: \"0c2852d0-4c85-4a25-8805-24cdeeca6f8c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.633523 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633441    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/94860428-39ed-41be-b019-fb867d18c0a7-klusterlet-config\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.633523 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.633457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.643099 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.643072    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94860428-39ed-41be-b019-fb867d18c0a7-tmp\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.643448 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.643429    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/94860428-39ed-41be-b019-fb867d18c0a7-klusterlet-config\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.648070 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.648041    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg88n\" (UniqueName: \"kubernetes.io/projected/94860428-39ed-41be-b019-fb867d18c0a7-kube-api-access-mg88n\") pod \"klusterlet-addon-workmgr-54b45b7555-8w7gx\" (UID: \"94860428-39ed-41be-b019-fb867d18c0a7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.734162 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734067    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-hub\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.734162 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734111    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2852d0-4c85-4a25-8805-24cdeeca6f8c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll\" (UID: \"0c2852d0-4c85-4a25-8805-24cdeeca6f8c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.734345 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734269    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/68751f3d-01e1-4203-a68f-6282bba51534-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.734345 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734307    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jbn9q\" (UniqueName: \"kubernetes.io/projected/0c2852d0-4c85-4a25-8805-24cdeeca6f8c-kube-api-access-jbn9q\") pod \"managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll\" (UID: \"0c2852d0-4c85-4a25-8805-24cdeeca6f8c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.734412 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734344    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.734461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734418    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-ca\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.734461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734445    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.734594 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.734472    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jffkd\" (UniqueName: \"kubernetes.io/projected/68751f3d-01e1-4203-a68f-6282bba51534-kube-api-access-jffkd\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.735066 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.735041    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/68751f3d-01e1-4203-a68f-6282bba51534-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.736719 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.736695    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-hub\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.736884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.736865    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c2852d0-4c85-4a25-8805-24cdeeca6f8c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll\" (UID: \"0c2852d0-4c85-4a25-8805-24cdeeca6f8c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.736945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.736897    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.737079 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.737064    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-ca\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.737180 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.737161    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68751f3d-01e1-4203-a68f-6282bba51534-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.742673 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.742644    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbn9q\" (UniqueName: \"kubernetes.io/projected/0c2852d0-4c85-4a25-8805-24cdeeca6f8c-kube-api-access-jbn9q\") pod \"managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll\" (UID: \"0c2852d0-4c85-4a25-8805-24cdeeca6f8c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.742810 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.742782    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffkd\" (UniqueName: \"kubernetes.io/projected/68751f3d-01e1-4203-a68f-6282bba51534-kube-api-access-jffkd\") pod \"cluster-proxy-proxy-agent-5dfb6dc495-jsvf6\" (UID: \"68751f3d-01e1-4203-a68f-6282bba51534\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.744442 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.744421    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:02.773082 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.773050    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"
May 25 11:18:02.779857 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.779826    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:18:02.949310 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.949278    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll"]
May 25 11:18:02.953150 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:18:02.953104    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c2852d0_4c85_4a25_8805_24cdeeca6f8c.slice/crio-435de76191c834141c35f49f5d8cd25fa7d40b101caee044ad58d498721b0a2b WatchSource:0}: Error finding container 435de76191c834141c35f49f5d8cd25fa7d40b101caee044ad58d498721b0a2b: Status 404 returned error can't find the container with id 435de76191c834141c35f49f5d8cd25fa7d40b101caee044ad58d498721b0a2b
May 25 11:18:02.954332 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.954311    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"]
May 25 11:18:02.956154 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:18:02.956133    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94860428_39ed_41be_b019_fb867d18c0a7.slice/crio-e2eaa3b16a36442d3688ef9e15b2cdb99f81941670683104b11d3c059b86389c WatchSource:0}: Error finding container e2eaa3b16a36442d3688ef9e15b2cdb99f81941670683104b11d3c059b86389c: Status 404 returned error can't find the container with id e2eaa3b16a36442d3688ef9e15b2cdb99f81941670683104b11d3c059b86389c
May 25 11:18:02.964635 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:02.964583    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"]
May 25 11:18:02.968007 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:18:02.967979    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68751f3d_01e1_4203_a68f_6282bba51534.slice/crio-980dcfa1dcbd7c7e77b8f1eabda2da5753303e288404fb332a660e2b482e8452 WatchSource:0}: Error finding container 980dcfa1dcbd7c7e77b8f1eabda2da5753303e288404fb332a660e2b482e8452: Status 404 returned error can't find the container with id 980dcfa1dcbd7c7e77b8f1eabda2da5753303e288404fb332a660e2b482e8452
May 25 11:18:03.059255 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:03.059167    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" event={"ID":"68751f3d-01e1-4203-a68f-6282bba51534","Type":"ContainerStarted","Data":"980dcfa1dcbd7c7e77b8f1eabda2da5753303e288404fb332a660e2b482e8452"}
May 25 11:18:03.060162 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:03.060133    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll" event={"ID":"0c2852d0-4c85-4a25-8805-24cdeeca6f8c","Type":"ContainerStarted","Data":"435de76191c834141c35f49f5d8cd25fa7d40b101caee044ad58d498721b0a2b"}
May 25 11:18:03.061038 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:03.061015    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx" event={"ID":"94860428-39ed-41be-b019-fb867d18c0a7","Type":"ContainerStarted","Data":"e2eaa3b16a36442d3688ef9e15b2cdb99f81941670683104b11d3c059b86389c"}
May 25 11:18:03.441509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:03.441471    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:03.441555    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:03.441578    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:03.441651    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:03.441681    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:03.441691    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:03.441721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:11.441704688 +0000 UTC m=+50.196681461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:03.441729    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:03.441735    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:11.441729415 +0000 UTC m=+50.196706188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:18:03.441971 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:03.441795    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:11.441777162 +0000 UTC m=+50.196753936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:18:07.275875 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:07.275836    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:18:07.278510 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:07.278483    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6275e60c-94fe-4622-a186-d02cea5494d0-original-pull-secret\") pod \"global-pull-secret-syncer-8smhx\" (UID: \"6275e60c-94fe-4622-a186-d02cea5494d0\") " pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:18:07.545255 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:07.545105    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8smhx"
May 25 11:18:10.769595 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:10.769311    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8smhx"]
May 25 11:18:10.772777 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:18:10.772732    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6275e60c_94fe_4622_a186_d02cea5494d0.slice/crio-a3964863ebca64c6012e45d95f560554caf551f368190cce034438f4e2dd4ca1 WatchSource:0}: Error finding container a3964863ebca64c6012e45d95f560554caf551f368190cce034438f4e2dd4ca1: Status 404 returned error can't find the container with id a3964863ebca64c6012e45d95f560554caf551f368190cce034438f4e2dd4ca1
May 25 11:18:11.079343 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.079303    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx" event={"ID":"94860428-39ed-41be-b019-fb867d18c0a7","Type":"ContainerStarted","Data":"6cfd00fed5e2584f008986658a167bb065c4ff1c2e458a05dfae8a09fb50a2b2"}
May 25 11:18:11.079616 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.079570    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:11.080568 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.080522    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8smhx" event={"ID":"6275e60c-94fe-4622-a186-d02cea5494d0","Type":"ContainerStarted","Data":"a3964863ebca64c6012e45d95f560554caf551f368190cce034438f4e2dd4ca1"}
May 25 11:18:11.081505 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.081482    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:18:11.081917 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.081893    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll" event={"ID":"0c2852d0-4c85-4a25-8805-24cdeeca6f8c","Type":"ContainerStarted","Data":"56d5244c0ed9588c8b8523f37411e88bc962854fa8ef96288d43c646a137c7b3"}
May 25 11:18:11.083226 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.083206    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" event={"ID":"68751f3d-01e1-4203-a68f-6282bba51534","Type":"ContainerStarted","Data":"7ac097256f9e8eb698128ce108b133748b72e0bd532995fb32cbf44df4ecbb5b"}
May 25 11:18:11.098710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.098662    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx" podStartSLOduration=1.414154784 podStartE2EDuration="9.098645364s" podCreationTimestamp="2026-05-25 11:18:02 +0000 UTC" firstStartedPulling="2026-05-25 11:18:02.95779905 +0000 UTC m=+41.712775825" lastFinishedPulling="2026-05-25 11:18:10.642289628 +0000 UTC m=+49.397266405" observedRunningTime="2026-05-25 11:18:11.098322197 +0000 UTC m=+49.853298994" watchObservedRunningTime="2026-05-25 11:18:11.098645364 +0000 UTC m=+49.853622163"
May 25 11:18:11.511077 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.511039    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:18:11.511285 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.511091    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:18:11.511285 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.511169    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:18:11.511285 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:11.511200    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:11.511285 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:11.511217    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:18:11.511285 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:11.511245    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:11.511285 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:11.511198    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:11.511617 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:11.511274    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.51126026 +0000 UTC m=+66.266237041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:18:11.511617 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:11.511323    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.511306058 +0000 UTC m=+66.266282840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:18:11.511617 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:11.511342    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.511333157 +0000 UTC m=+66.266309931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:18:11.856278 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:11.856210    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll" podStartSLOduration=2.231858188 podStartE2EDuration="9.85619117s" podCreationTimestamp="2026-05-25 11:18:02 +0000 UTC" firstStartedPulling="2026-05-25 11:18:02.955413765 +0000 UTC m=+41.710390542" lastFinishedPulling="2026-05-25 11:18:10.579746733 +0000 UTC m=+49.334723524" observedRunningTime="2026-05-25 11:18:11.136748422 +0000 UTC m=+49.891725219" watchObservedRunningTime="2026-05-25 11:18:11.85619117 +0000 UTC m=+50.611167967"
May 25 11:18:15.098741 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:15.098699    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" event={"ID":"68751f3d-01e1-4203-a68f-6282bba51534","Type":"ContainerStarted","Data":"33a1f8239dc689db72c7276d3a9d7d3175e8738440255f89723a1d34bd048c18"}
May 25 11:18:15.099192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:15.098746    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" event={"ID":"68751f3d-01e1-4203-a68f-6282bba51534","Type":"ContainerStarted","Data":"fd8084c83d5a604286c236787a1fc6e70fdefb0dc23414ce8f00ba37b8e3b24b"}
May 25 11:18:15.099938 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:15.099913    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8smhx" event={"ID":"6275e60c-94fe-4622-a186-d02cea5494d0","Type":"ContainerStarted","Data":"cb131c45dbeace07bcd875e61b8410eb477132d72eacb2ab4fea9d61924bc270"}
May 25 11:18:15.116246 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:15.116204    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" podStartSLOduration=1.363887952 podStartE2EDuration="13.11618969s" podCreationTimestamp="2026-05-25 11:18:02 +0000 UTC" firstStartedPulling="2026-05-25 11:18:02.969688691 +0000 UTC m=+41.724665465" lastFinishedPulling="2026-05-25 11:18:14.721990426 +0000 UTC m=+53.476967203" observedRunningTime="2026-05-25 11:18:15.115837184 +0000 UTC m=+53.870813973" watchObservedRunningTime="2026-05-25 11:18:15.11618969 +0000 UTC m=+53.871166484"
May 25 11:18:15.129598 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:15.129531    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8smhx" podStartSLOduration=36.172980866 podStartE2EDuration="40.129516432s" podCreationTimestamp="2026-05-25 11:17:35 +0000 UTC" firstStartedPulling="2026-05-25 11:18:10.775118491 +0000 UTC m=+49.530095279" lastFinishedPulling="2026-05-25 11:18:14.731654067 +0000 UTC m=+53.486630845" observedRunningTime="2026-05-25 11:18:15.128719776 +0000 UTC m=+53.883696572" watchObservedRunningTime="2026-05-25 11:18:15.129516432 +0000 UTC m=+53.884493229"
May 25 11:18:27.540318 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.540273    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.540337    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.540368    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.540420    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.540467    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.540478    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.540491    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:59.540474708 +0000 UTC m=+98.295451481 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.540420    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.540520    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:59.54050527 +0000 UTC m=+98.295482045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:18:27.540807 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.540563    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:59.540533865 +0000 UTC m=+98.295510641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:18:27.641432 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.641394    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:18:27.641557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.641497    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:18:27.645174 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.645157    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:27.645217 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.645165    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:27.651990 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.651962    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 11:18:27.652119 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:27.652055    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:19:31.652033729 +0000 UTC m=+130.407010520 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : secret "metrics-daemon-secret" not found
May 25 11:18:27.654478 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.654459    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:27.664721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.664693    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwvm\" (UniqueName: \"kubernetes.io/projected/2397ea2a-dfea-45e4-8858-e51400496bff-kube-api-access-zcwvm\") pod \"network-check-target-dcdqx\" (UID: \"2397ea2a-dfea-45e4-8858-e51400496bff\") " pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:18:27.953639 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.953609    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j5m5s\""
May 25 11:18:27.961244 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:27.961225    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:18:28.082699 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:28.082660    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dcdqx"]
May 25 11:18:28.086419 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:18:28.086383    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2397ea2a_dfea_45e4_8858_e51400496bff.slice/crio-b087a1e395d81f9a1ef3e8705bf9887f161182eea66be7ba9c9f1b86cae55484 WatchSource:0}: Error finding container b087a1e395d81f9a1ef3e8705bf9887f161182eea66be7ba9c9f1b86cae55484: Status 404 returned error can't find the container with id b087a1e395d81f9a1ef3e8705bf9887f161182eea66be7ba9c9f1b86cae55484
May 25 11:18:28.134560 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:28.134510    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dcdqx" event={"ID":"2397ea2a-dfea-45e4-8858-e51400496bff","Type":"ContainerStarted","Data":"b087a1e395d81f9a1ef3e8705bf9887f161182eea66be7ba9c9f1b86cae55484"}
May 25 11:18:31.142285 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:31.142246    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dcdqx" event={"ID":"2397ea2a-dfea-45e4-8858-e51400496bff","Type":"ContainerStarted","Data":"92db0d1d0b859aa19a790f730b03e0faf242adf6fdca0d49de29320f6ac1630b"}
May 25 11:18:31.142720 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:31.142482    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:18:31.162652 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:31.162598    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dcdqx" podStartSLOduration=67.677487849 podStartE2EDuration="1m10.16258008s" podCreationTimestamp="2026-05-25 11:17:21 +0000 UTC" firstStartedPulling="2026-05-25 11:18:28.088244572 +0000 UTC m=+66.843221346" lastFinishedPulling="2026-05-25 11:18:30.5733368 +0000 UTC m=+69.328313577" observedRunningTime="2026-05-25 11:18:31.161232396 +0000 UTC m=+69.916209193" watchObservedRunningTime="2026-05-25 11:18:31.16258008 +0000 UTC m=+69.917556879"
May 25 11:18:59.595251 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:59.595200    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:59.595289    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:18:59.595323    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:59.595351    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:59.595425    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:59.595467    2566 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:59.595429    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert podName:608f8a96-f300-4efe-89d2-51ca889a3e67 nodeName:}" failed. No retries permitted until 2026-05-25 11:20:03.595414099 +0000 UTC m=+162.350390873 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert") pod "ingress-canary-tnpqg" (UID: "608f8a96-f300-4efe-89d2-51ca889a3e67") : secret "canary-serving-cert" not found
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:59.595482    2566 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-779dd5d66c-dw82q: secret "image-registry-tls" not found
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:59.595500    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls podName:26513b67-e7e4-43c9-8a09-d496226e067e nodeName:}" failed. No retries permitted until 2026-05-25 11:20:03.59548337 +0000 UTC m=+162.350460144 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls") pod "dns-default-d7gl7" (UID: "26513b67-e7e4-43c9-8a09-d496226e067e") : secret "dns-default-metrics-tls" not found
May 25 11:18:59.595728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:18:59.595568    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls podName:629466b0-845b-4707-8e9d-ad1dd1e2df05 nodeName:}" failed. No retries permitted until 2026-05-25 11:20:03.595531831 +0000 UTC m=+162.350508622 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls") pod "image-registry-779dd5d66c-dw82q" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05") : secret "image-registry-tls" not found
May 25 11:19:02.146234 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:02.146201    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dcdqx"
May 25 11:19:30.666420 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:30.666389    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jqzls_7ad17068-4f7d-491f-8d77-61c903d241cb/dns-node-resolver/0.log"
May 25 11:19:31.738307 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:31.738254    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:19:31.738750 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:19:31.738401    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 11:19:31.738750 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:19:31.738479    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs podName:61545c61-84b8-432a-9443-e9fd8b59060f nodeName:}" failed. No retries permitted until 2026-05-25 11:21:33.738462026 +0000 UTC m=+252.493438801 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs") pod "network-metrics-daemon-xm9qk" (UID: "61545c61-84b8-432a-9443-e9fd8b59060f") : secret "metrics-daemon-secret" not found
May 25 11:19:32.066477 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:32.066396    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z7trl_d14d47c1-872a-4915-9cc5-f57cf19c52bb/node-ca/0.log"
May 25 11:19:42.781458 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:42.781399    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" podUID="68751f3d-01e1-4203-a68f-6282bba51534" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 11:19:50.320174 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.320135    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qh96l"]
May 25 11:19:50.323139 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.323123    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.326024 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.326002    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 11:19:50.327027 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.327005    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 11:19:50.327167 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.327030    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 11:19:50.327414 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.327396    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rgvrv\""
May 25 11:19:50.327497 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.327416    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 11:19:50.348888 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.348858    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qh96l"]
May 25 11:19:50.481590 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.481532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b44825ca-8493-4c51-9956-1d2f5137d19f-crio-socket\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.481590 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.481592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b44825ca-8493-4c51-9956-1d2f5137d19f-data-volume\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.481795 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.481720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mx2\" (UniqueName: \"kubernetes.io/projected/b44825ca-8493-4c51-9956-1d2f5137d19f-kube-api-access-s4mx2\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.481795 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.481748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b44825ca-8493-4c51-9956-1d2f5137d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.481795 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.481784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b44825ca-8493-4c51-9956-1d2f5137d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.582570 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.582449    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b44825ca-8493-4c51-9956-1d2f5137d19f-crio-socket\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.582570 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.582493    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b44825ca-8493-4c51-9956-1d2f5137d19f-data-volume\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.582782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.582595    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mx2\" (UniqueName: \"kubernetes.io/projected/b44825ca-8493-4c51-9956-1d2f5137d19f-kube-api-access-s4mx2\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.582782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.582609    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b44825ca-8493-4c51-9956-1d2f5137d19f-crio-socket\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.582782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.582624    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b44825ca-8493-4c51-9956-1d2f5137d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.582782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.582682    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b44825ca-8493-4c51-9956-1d2f5137d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.582964 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.582934    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b44825ca-8493-4c51-9956-1d2f5137d19f-data-volume\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.583135 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.583120    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b44825ca-8493-4c51-9956-1d2f5137d19f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.584995 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.584976    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b44825ca-8493-4c51-9956-1d2f5137d19f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.594230 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.594203    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mx2\" (UniqueName: \"kubernetes.io/projected/b44825ca-8493-4c51-9956-1d2f5137d19f-kube-api-access-s4mx2\") pod \"insights-runtime-extractor-qh96l\" (UID: \"b44825ca-8493-4c51-9956-1d2f5137d19f\") " pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.631936 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.631896    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qh96l"
May 25 11:19:50.769605 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:50.769567    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qh96l"]
May 25 11:19:50.772960 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:19:50.772924    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44825ca_8493_4c51_9956_1d2f5137d19f.slice/crio-638d31cd94e8b671de04a5f38c6f6a25bc28251de4317f3de5e45295159303fe WatchSource:0}: Error finding container 638d31cd94e8b671de04a5f38c6f6a25bc28251de4317f3de5e45295159303fe: Status 404 returned error can't find the container with id 638d31cd94e8b671de04a5f38c6f6a25bc28251de4317f3de5e45295159303fe
May 25 11:19:51.323452 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:51.323407    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qh96l" event={"ID":"b44825ca-8493-4c51-9956-1d2f5137d19f","Type":"ContainerStarted","Data":"c9707adfc22981cc3621e658eee8f000d6c527b0f7ab12ea216aa57f00ca8ff8"}
May 25 11:19:51.323452 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:51.323449    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qh96l" event={"ID":"b44825ca-8493-4c51-9956-1d2f5137d19f","Type":"ContainerStarted","Data":"638d31cd94e8b671de04a5f38c6f6a25bc28251de4317f3de5e45295159303fe"}
May 25 11:19:52.327882 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:52.327844    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qh96l" event={"ID":"b44825ca-8493-4c51-9956-1d2f5137d19f","Type":"ContainerStarted","Data":"f5e9b7cd78324dee3fc76f49828adde8d63662e92fa38dde01610b3b69cdff36"}
May 25 11:19:52.781415 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:52.781376    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" podUID="68751f3d-01e1-4203-a68f-6282bba51534" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 11:19:53.331683 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:53.331643    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qh96l" event={"ID":"b44825ca-8493-4c51-9956-1d2f5137d19f","Type":"ContainerStarted","Data":"aca76477f518777aad30a45273da9edd59058734388f1633207debafdfd42775"}
May 25 11:19:53.348974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:53.348918    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qh96l" podStartSLOduration=1.274546535 podStartE2EDuration="3.348904151s" podCreationTimestamp="2026-05-25 11:19:50 +0000 UTC" firstStartedPulling="2026-05-25 11:19:50.851571853 +0000 UTC m=+149.606548628" lastFinishedPulling="2026-05-25 11:19:52.925929461 +0000 UTC m=+151.680906244" observedRunningTime="2026-05-25 11:19:53.348022099 +0000 UTC m=+152.102998898" watchObservedRunningTime="2026-05-25 11:19:53.348904151 +0000 UTC m=+152.103880946"
May 25 11:19:58.616402 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:19:58.616343    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" podUID="629466b0-845b-4707-8e9d-ad1dd1e2df05"
May 25 11:19:58.627493 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:19:58.627447    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d7gl7" podUID="26513b67-e7e4-43c9-8a09-d496226e067e"
May 25 11:19:58.646789 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:19:58.646750    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-tnpqg" podUID="608f8a96-f300-4efe-89d2-51ca889a3e67"
May 25 11:19:58.856758 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:19:58.856715    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xm9qk" podUID="61545c61-84b8-432a-9443-e9fd8b59060f"
May 25 11:19:59.347504 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:59.347474    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7gl7"
May 25 11:19:59.347694 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:19:59.347474    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:02.781413 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:02.781375    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" podUID="68751f3d-01e1-4203-a68f-6282bba51534" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 11:20:02.781875 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:02.781453    2566 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6"
May 25 11:20:02.781957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:02.781928    2566 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"33a1f8239dc689db72c7276d3a9d7d3175e8738440255f89723a1d34bd048c18"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 25 11:20:02.781998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:02.781977    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" podUID="68751f3d-01e1-4203-a68f-6282bba51534" containerName="service-proxy" containerID="cri-o://33a1f8239dc689db72c7276d3a9d7d3175e8738440255f89723a1d34bd048c18" gracePeriod=30
May 25 11:20:03.358345 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.358309    2566 generic.go:358] "Generic (PLEG): container finished" podID="68751f3d-01e1-4203-a68f-6282bba51534" containerID="33a1f8239dc689db72c7276d3a9d7d3175e8738440255f89723a1d34bd048c18" exitCode=2
May 25 11:20:03.358518 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.358380    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" event={"ID":"68751f3d-01e1-4203-a68f-6282bba51534","Type":"ContainerDied","Data":"33a1f8239dc689db72c7276d3a9d7d3175e8738440255f89723a1d34bd048c18"}
May 25 11:20:03.358518 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.358426    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dfb6dc495-jsvf6" event={"ID":"68751f3d-01e1-4203-a68f-6282bba51534","Type":"ContainerStarted","Data":"bed170dbfe09a6d81231490c9b44c0f5204fa091cdf7f033781a27015ad24bac"}
May 25 11:20:03.691171 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.691046    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:20:03.691171 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.691129    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:20:03.691171 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.691158    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:03.693566 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.693517    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26513b67-e7e4-43c9-8a09-d496226e067e-metrics-tls\") pod \"dns-default-d7gl7\" (UID: \"26513b67-e7e4-43c9-8a09-d496226e067e\") " pod="openshift-dns/dns-default-d7gl7"
May 25 11:20:03.693697 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.693594    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"image-registry-779dd5d66c-dw82q\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") " pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:03.693697 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.693605    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/608f8a96-f300-4efe-89d2-51ca889a3e67-cert\") pod \"ingress-canary-tnpqg\" (UID: \"608f8a96-f300-4efe-89d2-51ca889a3e67\") " pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:20:03.852613 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.852580    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t8nfn\""
May 25 11:20:03.852613 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.852588    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-srlsx\""
May 25 11:20:03.858603 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.858569    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:03.858603 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.858606    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7gl7"
May 25 11:20:03.989782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:03.989750    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-779dd5d66c-dw82q"]
May 25 11:20:03.992993 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:20:03.992959    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629466b0_845b_4707_8e9d_ad1dd1e2df05.slice/crio-6416e2e07f8902b6f99578caf872d3bd5f859a2937c55fac9eff05d3a1f6a4e4 WatchSource:0}: Error finding container 6416e2e07f8902b6f99578caf872d3bd5f859a2937c55fac9eff05d3a1f6a4e4: Status 404 returned error can't find the container with id 6416e2e07f8902b6f99578caf872d3bd5f859a2937c55fac9eff05d3a1f6a4e4
May 25 11:20:04.002489 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.002441    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7gl7"]
May 25 11:20:04.005724 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:20:04.005700    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26513b67_e7e4_43c9_8a09_d496226e067e.slice/crio-28d12eeae3c551b6911dec44acc3edcaeebbbe645a11330adac05a6c4805d570 WatchSource:0}: Error finding container 28d12eeae3c551b6911dec44acc3edcaeebbbe645a11330adac05a6c4805d570: Status 404 returned error can't find the container with id 28d12eeae3c551b6911dec44acc3edcaeebbbe645a11330adac05a6c4805d570
May 25 11:20:04.362354 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.362311    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7gl7" event={"ID":"26513b67-e7e4-43c9-8a09-d496226e067e","Type":"ContainerStarted","Data":"28d12eeae3c551b6911dec44acc3edcaeebbbe645a11330adac05a6c4805d570"}
May 25 11:20:04.363737 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.363711    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" event={"ID":"629466b0-845b-4707-8e9d-ad1dd1e2df05","Type":"ContainerStarted","Data":"ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6"}
May 25 11:20:04.363861 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.363743    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" event={"ID":"629466b0-845b-4707-8e9d-ad1dd1e2df05","Type":"ContainerStarted","Data":"6416e2e07f8902b6f99578caf872d3bd5f859a2937c55fac9eff05d3a1f6a4e4"}
May 25 11:20:04.363861 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.363856    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:04.392306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.392249    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" podStartSLOduration=162.392228662 podStartE2EDuration="2m42.392228662s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:20:04.391390231 +0000 UTC m=+163.146367027" watchObservedRunningTime="2026-05-25 11:20:04.392228662 +0000 UTC m=+163.147205463"
May 25 11:20:04.502741 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.502710    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ss9q2"]
May 25 11:20:04.505303 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.505279    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.507756 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.507734    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jfmc7\""
May 25 11:20:04.507963 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.507939    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 11:20:04.508076 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.507997    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 11:20:04.508207 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.508190    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 11:20:04.508599 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.508581    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 11:20:04.508704 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.508685    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 11:20:04.509463 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.509443    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 11:20:04.599753 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-wtmp\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.599960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-tls\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.599960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.599960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-root\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.599960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-sys\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.599960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99202225-27af-4ec5-8722-ee88f742653b-metrics-client-ca\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.599960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4p2\" (UniqueName: \"kubernetes.io/projected/99202225-27af-4ec5-8722-ee88f742653b-kube-api-access-4b4p2\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.599960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.599951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-textfile\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.600315 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.600015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700472 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700383    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700472 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700429    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-root\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700472 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700452    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-sys\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700479    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99202225-27af-4ec5-8722-ee88f742653b-metrics-client-ca\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700511    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4p2\" (UniqueName: \"kubernetes.io/projected/99202225-27af-4ec5-8722-ee88f742653b-kube-api-access-4b4p2\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700517    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-root\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700559    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-textfile\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700595    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-sys\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700661    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700707    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-wtmp\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.700775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700743    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-tls\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.701172 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:20:04.700862    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 25 11:20:04.701172 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:20:04.700920    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-tls podName:99202225-27af-4ec5-8722-ee88f742653b nodeName:}" failed. No retries permitted until 2026-05-25 11:20:05.200901224 +0000 UTC m=+163.955878001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-tls") pod "node-exporter-ss9q2" (UID: "99202225-27af-4ec5-8722-ee88f742653b") : secret "node-exporter-tls" not found
May 25 11:20:04.701172 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700947    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-textfile\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.701172 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.700980    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-wtmp\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.701378 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.701282    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99202225-27af-4ec5-8722-ee88f742653b-metrics-client-ca\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.701378 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.701289    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.703189 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.703169    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:04.713030 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:04.713001    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4p2\" (UniqueName: \"kubernetes.io/projected/99202225-27af-4ec5-8722-ee88f742653b-kube-api-access-4b4p2\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:05.206627 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:05.206580    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-tls\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:05.209237 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:05.209211    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99202225-27af-4ec5-8722-ee88f742653b-node-exporter-tls\") pod \"node-exporter-ss9q2\" (UID: \"99202225-27af-4ec5-8722-ee88f742653b\") " pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:05.416790 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:05.416156    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ss9q2"
May 25 11:20:05.428840 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:20:05.428807    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99202225_27af_4ec5_8722_ee88f742653b.slice/crio-28281956948ae44c50a55aa89fda0e2725785ae06e9c60bcebab727edf5819f2 WatchSource:0}: Error finding container 28281956948ae44c50a55aa89fda0e2725785ae06e9c60bcebab727edf5819f2: Status 404 returned error can't find the container with id 28281956948ae44c50a55aa89fda0e2725785ae06e9c60bcebab727edf5819f2
May 25 11:20:06.372533 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:06.372496    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ss9q2" event={"ID":"99202225-27af-4ec5-8722-ee88f742653b","Type":"ContainerStarted","Data":"78b533a6c446b999dcb7908a3106e090cf9b30d601b58a129a6889ae44ff1571"}
May 25 11:20:06.372978 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:06.372559    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ss9q2" event={"ID":"99202225-27af-4ec5-8722-ee88f742653b","Type":"ContainerStarted","Data":"28281956948ae44c50a55aa89fda0e2725785ae06e9c60bcebab727edf5819f2"}
May 25 11:20:06.374064 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:06.374042    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7gl7" event={"ID":"26513b67-e7e4-43c9-8a09-d496226e067e","Type":"ContainerStarted","Data":"67971f636ba7a5dbe3572543b54a1ed8948e366497dfdc1ac1cbce987cad44f1"}
May 25 11:20:06.374064 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:06.374067    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7gl7" event={"ID":"26513b67-e7e4-43c9-8a09-d496226e067e","Type":"ContainerStarted","Data":"889142610b5a18b0a1c1b95047d606db13a0a888d66b11416bc6e4a38b2b0032"}
May 25 11:20:06.374206 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:06.374195    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d7gl7"
May 25 11:20:06.414159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:06.414103    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-d7gl7" podStartSLOduration=130.131406689 podStartE2EDuration="2m11.414089152s" podCreationTimestamp="2026-05-25 11:17:55 +0000 UTC" firstStartedPulling="2026-05-25 11:20:04.007619115 +0000 UTC m=+162.762595893" lastFinishedPulling="2026-05-25 11:20:05.290301571 +0000 UTC m=+164.045278356" observedRunningTime="2026-05-25 11:20:06.412877694 +0000 UTC m=+165.167854491" watchObservedRunningTime="2026-05-25 11:20:06.414089152 +0000 UTC m=+165.169065947"
May 25 11:20:07.377502 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:07.377425    2566 generic.go:358] "Generic (PLEG): container finished" podID="99202225-27af-4ec5-8722-ee88f742653b" containerID="78b533a6c446b999dcb7908a3106e090cf9b30d601b58a129a6889ae44ff1571" exitCode=0
May 25 11:20:07.377903 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:07.377502    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ss9q2" event={"ID":"99202225-27af-4ec5-8722-ee88f742653b","Type":"ContainerDied","Data":"78b533a6c446b999dcb7908a3106e090cf9b30d601b58a129a6889ae44ff1571"}
May 25 11:20:08.381904 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:08.381869    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ss9q2" event={"ID":"99202225-27af-4ec5-8722-ee88f742653b","Type":"ContainerStarted","Data":"685a148eb6384df9f06b3b6f511feeea3acee27b64a148a684a63954286e18b8"}
May 25 11:20:08.382313 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:08.381921    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ss9q2" event={"ID":"99202225-27af-4ec5-8722-ee88f742653b","Type":"ContainerStarted","Data":"b8b03c32fd6c55491eb5eb9e98df6586c4b7365beed420a5374823a1e65ff1df"}
May 25 11:20:08.403716 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:08.403651    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ss9q2" podStartSLOduration=3.567288591 podStartE2EDuration="4.403635561s" podCreationTimestamp="2026-05-25 11:20:04 +0000 UTC" firstStartedPulling="2026-05-25 11:20:05.431083854 +0000 UTC m=+164.186060627" lastFinishedPulling="2026-05-25 11:20:06.267430812 +0000 UTC m=+165.022407597" observedRunningTime="2026-05-25 11:20:08.402639014 +0000 UTC m=+167.157615813" watchObservedRunningTime="2026-05-25 11:20:08.403635561 +0000 UTC m=+167.158612356"
May 25 11:20:11.080125 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:11.080040    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx" podUID="94860428-39ed-41be-b019-fb867d18c0a7" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
May 25 11:20:11.393017 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:11.392985    2566 generic.go:358] "Generic (PLEG): container finished" podID="0c2852d0-4c85-4a25-8805-24cdeeca6f8c" containerID="56d5244c0ed9588c8b8523f37411e88bc962854fa8ef96288d43c646a137c7b3" exitCode=255
May 25 11:20:11.393187 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:11.393062    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll" event={"ID":"0c2852d0-4c85-4a25-8805-24cdeeca6f8c","Type":"ContainerDied","Data":"56d5244c0ed9588c8b8523f37411e88bc962854fa8ef96288d43c646a137c7b3"}
May 25 11:20:11.393447 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:11.393423    2566 scope.go:117] "RemoveContainer" containerID="56d5244c0ed9588c8b8523f37411e88bc962854fa8ef96288d43c646a137c7b3"
May 25 11:20:11.394320 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:11.394303    2566 generic.go:358] "Generic (PLEG): container finished" podID="94860428-39ed-41be-b019-fb867d18c0a7" containerID="6cfd00fed5e2584f008986658a167bb065c4ff1c2e458a05dfae8a09fb50a2b2" exitCode=1
May 25 11:20:11.394396 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:11.394342    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx" event={"ID":"94860428-39ed-41be-b019-fb867d18c0a7","Type":"ContainerDied","Data":"6cfd00fed5e2584f008986658a167bb065c4ff1c2e458a05dfae8a09fb50a2b2"}
May 25 11:20:11.394629 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:11.394616    2566 scope.go:117] "RemoveContainer" containerID="6cfd00fed5e2584f008986658a167bb065c4ff1c2e458a05dfae8a09fb50a2b2"
May 25 11:20:12.399418 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:12.399379    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx" event={"ID":"94860428-39ed-41be-b019-fb867d18c0a7","Type":"ContainerStarted","Data":"27016a505c8bb327143d079e09f7b8595c6e0a96662301e28b8704d90f68824d"}
May 25 11:20:12.399902 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:12.399741    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:20:12.400348 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:12.400329    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54b45b7555-8w7gx"
May 25 11:20:12.401007 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:12.400978    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64dfd8f5c6-hxnll" event={"ID":"0c2852d0-4c85-4a25-8805-24cdeeca6f8c","Type":"ContainerStarted","Data":"f97d6c74d7d9773b57e921505300a38b059b1564b43637af5d5e173cb4ef3dd8"}
May 25 11:20:13.834700 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:13.834673    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:20:13.835087 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:13.834679    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:20:13.836906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:13.836889    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjf4k\""
May 25 11:20:13.845705 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:13.845673    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnpqg"
May 25 11:20:13.976356 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:13.976319    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnpqg"]
May 25 11:20:13.979574 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:20:13.979532    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608f8a96_f300_4efe_89d2_51ca889a3e67.slice/crio-4a1a904c74c675fe35ad9c1de7aa6cb9e9f8a64d85148d53c98cf8682e9410a0 WatchSource:0}: Error finding container 4a1a904c74c675fe35ad9c1de7aa6cb9e9f8a64d85148d53c98cf8682e9410a0: Status 404 returned error can't find the container with id 4a1a904c74c675fe35ad9c1de7aa6cb9e9f8a64d85148d53c98cf8682e9410a0
May 25 11:20:14.408805 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:14.408772    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnpqg" event={"ID":"608f8a96-f300-4efe-89d2-51ca889a3e67","Type":"ContainerStarted","Data":"4a1a904c74c675fe35ad9c1de7aa6cb9e9f8a64d85148d53c98cf8682e9410a0"}
May 25 11:20:16.379482 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:16.379446    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d7gl7"
May 25 11:20:16.415269 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:16.415226    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnpqg" event={"ID":"608f8a96-f300-4efe-89d2-51ca889a3e67","Type":"ContainerStarted","Data":"779505fe39302b81632e8aa0126ab2f87e2b19eedca53aa1bd36e404ee8ed824"}
May 25 11:20:16.430666 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:16.430615    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tnpqg" podStartSLOduration=139.681349029 podStartE2EDuration="2m21.430598955s" podCreationTimestamp="2026-05-25 11:17:55 +0000 UTC" firstStartedPulling="2026-05-25 11:20:13.981407893 +0000 UTC m=+172.736384666" lastFinishedPulling="2026-05-25 11:20:15.730657818 +0000 UTC m=+174.485634592" observedRunningTime="2026-05-25 11:20:16.430114857 +0000 UTC m=+175.185091654" watchObservedRunningTime="2026-05-25 11:20:16.430598955 +0000 UTC m=+175.185575783"
May 25 11:20:22.524235 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:22.524200    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-779dd5d66c-dw82q"]
May 25 11:20:22.528177 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:22.528150    2566 patch_prober.go:28] interesting pod/image-registry-779dd5d66c-dw82q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 11:20:22.528304 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:22.528203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" podUID="629466b0-845b-4707-8e9d-ad1dd1e2df05" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:20:32.528530 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:32.528502    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:47.542644 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.542580    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" podUID="629466b0-845b-4707-8e9d-ad1dd1e2df05" containerName="registry" containerID="cri-o://ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6" gracePeriod=30
May 25 11:20:47.770088 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.770065    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:47.817002 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.816932    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.817002 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.816966    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-certificates\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.817002 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.816993    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-trusted-ca\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.817239 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.817017    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9dd2\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-kube-api-access-n9dd2\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.817767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.817506    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-image-registry-private-configuration\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.817767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.817591    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/629466b0-845b-4707-8e9d-ad1dd1e2df05-ca-trust-extracted\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.817767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.817585    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:47.817767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.817638    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-installation-pull-secrets\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.817767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.817685    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-bound-sa-token\") pod \"629466b0-845b-4707-8e9d-ad1dd1e2df05\" (UID: \"629466b0-845b-4707-8e9d-ad1dd1e2df05\") "
May 25 11:20:47.818112 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.817763    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:47.819445 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.819414    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-certificates\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:47.819608 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.819450    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/629466b0-845b-4707-8e9d-ad1dd1e2df05-trusted-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:47.823708 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.821400    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:47.823846 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.823803    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-kube-api-access-n9dd2" (OuterVolumeSpecName: "kube-api-access-n9dd2") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "kube-api-access-n9dd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:47.824140 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.824084    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:47.824491 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.824465    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:47.824600 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.824584    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:47.827910 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.827886    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629466b0-845b-4707-8e9d-ad1dd1e2df05-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "629466b0-845b-4707-8e9d-ad1dd1e2df05" (UID: "629466b0-845b-4707-8e9d-ad1dd1e2df05"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:20:47.920141 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.920109    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-image-registry-private-configuration\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:47.920141 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.920137    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/629466b0-845b-4707-8e9d-ad1dd1e2df05-ca-trust-extracted\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:47.920141 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.920147    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/629466b0-845b-4707-8e9d-ad1dd1e2df05-installation-pull-secrets\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:47.920371 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.920157    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-bound-sa-token\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:47.920371 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.920166    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-registry-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:47.920371 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:47.920174    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9dd2\" (UniqueName: \"kubernetes.io/projected/629466b0-845b-4707-8e9d-ad1dd1e2df05-kube-api-access-n9dd2\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:20:48.494848 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.494816    2566 generic.go:358] "Generic (PLEG): container finished" podID="629466b0-845b-4707-8e9d-ad1dd1e2df05" containerID="ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6" exitCode=0
May 25 11:20:48.495018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.494896    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q"
May 25 11:20:48.495018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.494892    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" event={"ID":"629466b0-845b-4707-8e9d-ad1dd1e2df05","Type":"ContainerDied","Data":"ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6"}
May 25 11:20:48.495018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.494928    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779dd5d66c-dw82q" event={"ID":"629466b0-845b-4707-8e9d-ad1dd1e2df05","Type":"ContainerDied","Data":"6416e2e07f8902b6f99578caf872d3bd5f859a2937c55fac9eff05d3a1f6a4e4"}
May 25 11:20:48.495018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.494951    2566 scope.go:117] "RemoveContainer" containerID="ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6"
May 25 11:20:48.502592 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.502554    2566 scope.go:117] "RemoveContainer" containerID="ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6"
May 25 11:20:48.502858 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:20:48.502834    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6\": container with ID starting with ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6 not found: ID does not exist" containerID="ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6"
May 25 11:20:48.502941 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.502866    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6"} err="failed to get container status \"ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6\": rpc error: code = NotFound desc = could not find container \"ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6\": container with ID starting with ea8257f5e5dee3140c6e3065f1ff25d847c8a7d9f45693c410e4cdc67cc02ea6 not found: ID does not exist"
May 25 11:20:48.511910 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.511885    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-779dd5d66c-dw82q"]
May 25 11:20:48.516463 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:48.516443    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-779dd5d66c-dw82q"]
May 25 11:20:49.834748 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:20:49.834717    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629466b0-845b-4707-8e9d-ad1dd1e2df05" path="/var/lib/kubelet/pods/629466b0-845b-4707-8e9d-ad1dd1e2df05/volumes"
May 25 11:21:33.766578 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:33.766501    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:21:33.768742 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:33.768719    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61545c61-84b8-432a-9443-e9fd8b59060f-metrics-certs\") pod \"network-metrics-daemon-xm9qk\" (UID: \"61545c61-84b8-432a-9443-e9fd8b59060f\") " pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:21:33.939611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:33.939575    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bl7fp\""
May 25 11:21:33.945363 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:33.945341    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xm9qk"
May 25 11:21:34.106885 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:34.106861    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xm9qk"]
May 25 11:21:34.109380 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:21:34.109352    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61545c61_84b8_432a_9443_e9fd8b59060f.slice/crio-bbcf2ffa32dd84532ad72b63ed476da61baa5803885c58a3c79a30f4e093f4cd WatchSource:0}: Error finding container bbcf2ffa32dd84532ad72b63ed476da61baa5803885c58a3c79a30f4e093f4cd: Status 404 returned error can't find the container with id bbcf2ffa32dd84532ad72b63ed476da61baa5803885c58a3c79a30f4e093f4cd
May 25 11:21:34.618396 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:34.618358    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xm9qk" event={"ID":"61545c61-84b8-432a-9443-e9fd8b59060f","Type":"ContainerStarted","Data":"bbcf2ffa32dd84532ad72b63ed476da61baa5803885c58a3c79a30f4e093f4cd"}
May 25 11:21:35.622554 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:35.622510    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xm9qk" event={"ID":"61545c61-84b8-432a-9443-e9fd8b59060f","Type":"ContainerStarted","Data":"24471efe25817622b02ccf223721e620875c9e98dc94b18316fc1841e63f5f31"}
May 25 11:21:35.622936 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:35.622560    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xm9qk" event={"ID":"61545c61-84b8-432a-9443-e9fd8b59060f","Type":"ContainerStarted","Data":"43c29df7bae0ad903a13ac1b9ca83faa2eae3f6cc0d85799371f189d435f4ba3"}
May 25 11:21:35.659623 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:21:35.659579    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xm9qk" podStartSLOduration=252.644631278 podStartE2EDuration="4m13.659565749s" podCreationTimestamp="2026-05-25 11:17:22 +0000 UTC" firstStartedPulling="2026-05-25 11:21:34.111175652 +0000 UTC m=+252.866152426" lastFinishedPulling="2026-05-25 11:21:35.126110121 +0000 UTC m=+253.881086897" observedRunningTime="2026-05-25 11:21:35.658974731 +0000 UTC m=+254.413951529" watchObservedRunningTime="2026-05-25 11:21:35.659565749 +0000 UTC m=+254.414542545"
May 25 11:22:21.721926 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:21.721898    2566 kubelet.go:1628] "Image garbage collection succeeded"
May 25 11:22:35.231782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.231752    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"]
May 25 11:22:35.234180 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.231979    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629466b0-845b-4707-8e9d-ad1dd1e2df05" containerName="registry"
May 25 11:22:35.234180 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.231992    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="629466b0-845b-4707-8e9d-ad1dd1e2df05" containerName="registry"
May 25 11:22:35.234180 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.232032    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="629466b0-845b-4707-8e9d-ad1dd1e2df05" containerName="registry"
May 25 11:22:35.235026 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.235011    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.237690 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.237664    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:22:35.237855 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.237841    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:22:35.238510 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.238492    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rvpn2\""
May 25 11:22:35.254172 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.254130    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"]
May 25 11:22:35.280591 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.280531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vb6q\" (UniqueName: \"kubernetes.io/projected/79eadbef-9fe5-41b3-8c64-f9404609925b-kube-api-access-5vb6q\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.280736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.280637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.280736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.280667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.381494 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.381449    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.381494 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.381494    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.381711 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.381532    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5vb6q\" (UniqueName: \"kubernetes.io/projected/79eadbef-9fe5-41b3-8c64-f9404609925b-kube-api-access-5vb6q\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.381957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.381933    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.381995 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.381943    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.390944 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.390913    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vb6q\" (UniqueName: \"kubernetes.io/projected/79eadbef-9fe5-41b3-8c64-f9404609925b-kube-api-access-5vb6q\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.544304 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.544203    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:35.676239 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.676195    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"]
May 25 11:22:35.679036 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:22:35.679010    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79eadbef_9fe5_41b3_8c64_f9404609925b.slice/crio-9ad62f8d22699c746c47af990618b51caf87e795cfa1e62b04190472825adcb6 WatchSource:0}: Error finding container 9ad62f8d22699c746c47af990618b51caf87e795cfa1e62b04190472825adcb6: Status 404 returned error can't find the container with id 9ad62f8d22699c746c47af990618b51caf87e795cfa1e62b04190472825adcb6
May 25 11:22:35.680867 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.680846    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:22:35.781978 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:35.781939    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84" event={"ID":"79eadbef-9fe5-41b3-8c64-f9404609925b","Type":"ContainerStarted","Data":"9ad62f8d22699c746c47af990618b51caf87e795cfa1e62b04190472825adcb6"}
May 25 11:22:41.799395 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:41.799299    2566 generic.go:358] "Generic (PLEG): container finished" podID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerID="35345e3d02832c1be2024d05192aebcaa7d0409c557625bbf72becd1e8326489" exitCode=0
May 25 11:22:41.799395 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:41.799342    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84" event={"ID":"79eadbef-9fe5-41b3-8c64-f9404609925b","Type":"ContainerDied","Data":"35345e3d02832c1be2024d05192aebcaa7d0409c557625bbf72becd1e8326489"}
May 25 11:22:44.808601 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:44.808562    2566 generic.go:358] "Generic (PLEG): container finished" podID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerID="cfc6930ffb3dc559e1df51cdfcca12f262fc077adacd30da384719cb5e93acf9" exitCode=0
May 25 11:22:44.808985 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:44.808661    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84" event={"ID":"79eadbef-9fe5-41b3-8c64-f9404609925b","Type":"ContainerDied","Data":"cfc6930ffb3dc559e1df51cdfcca12f262fc077adacd30da384719cb5e93acf9"}
May 25 11:22:51.829977 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:51.829937    2566 generic.go:358] "Generic (PLEG): container finished" podID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerID="7fb908ea870c33c47d7feb59fb8a435f5e119fc8421c7f5436ccdc5e06e3140f" exitCode=0
May 25 11:22:51.830362 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:51.830014    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84" event={"ID":"79eadbef-9fe5-41b3-8c64-f9404609925b","Type":"ContainerDied","Data":"7fb908ea870c33c47d7feb59fb8a435f5e119fc8421c7f5436ccdc5e06e3140f"}
May 25 11:22:52.953105 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:52.953075    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:22:53.020960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.020931    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-bundle\") pod \"79eadbef-9fe5-41b3-8c64-f9404609925b\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") "
May 25 11:22:53.021147 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.020974    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vb6q\" (UniqueName: \"kubernetes.io/projected/79eadbef-9fe5-41b3-8c64-f9404609925b-kube-api-access-5vb6q\") pod \"79eadbef-9fe5-41b3-8c64-f9404609925b\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") "
May 25 11:22:53.021147 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.021004    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-util\") pod \"79eadbef-9fe5-41b3-8c64-f9404609925b\" (UID: \"79eadbef-9fe5-41b3-8c64-f9404609925b\") "
May 25 11:22:53.021556 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.021514    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-bundle" (OuterVolumeSpecName: "bundle") pod "79eadbef-9fe5-41b3-8c64-f9404609925b" (UID: "79eadbef-9fe5-41b3-8c64-f9404609925b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:22:53.023092 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.023070    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79eadbef-9fe5-41b3-8c64-f9404609925b-kube-api-access-5vb6q" (OuterVolumeSpecName: "kube-api-access-5vb6q") pod "79eadbef-9fe5-41b3-8c64-f9404609925b" (UID: "79eadbef-9fe5-41b3-8c64-f9404609925b"). InnerVolumeSpecName "kube-api-access-5vb6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:22:53.025037 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.025013    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-util" (OuterVolumeSpecName: "util") pod "79eadbef-9fe5-41b3-8c64-f9404609925b" (UID: "79eadbef-9fe5-41b3-8c64-f9404609925b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:22:53.122026 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.121997    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:22:53.122026 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.122026    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vb6q\" (UniqueName: \"kubernetes.io/projected/79eadbef-9fe5-41b3-8c64-f9404609925b-kube-api-access-5vb6q\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:22:53.122204 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.122037    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eadbef-9fe5-41b3-8c64-f9404609925b-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:22:53.836068 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.836033    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84" event={"ID":"79eadbef-9fe5-41b3-8c64-f9404609925b","Type":"ContainerDied","Data":"9ad62f8d22699c746c47af990618b51caf87e795cfa1e62b04190472825adcb6"}
May 25 11:22:53.836068 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.836069    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad62f8d22699c746c47af990618b51caf87e795cfa1e62b04190472825adcb6"
May 25 11:22:53.836264 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:22:53.836113    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9qm84"
May 25 11:23:02.490582 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490534    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"]
May 25 11:23:02.490940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490777    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerName="util"
May 25 11:23:02.490940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490787    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerName="util"
May 25 11:23:02.490940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490794    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerName="pull"
May 25 11:23:02.490940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490799    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerName="pull"
May 25 11:23:02.490940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490806    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerName="extract"
May 25 11:23:02.490940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490811    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerName="extract"
May 25 11:23:02.490940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.490846    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="79eadbef-9fe5-41b3-8c64-f9404609925b" containerName="extract"
May 25 11:23:02.492735 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.492719    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.497384 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.497358    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 11:23:02.497528 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.497410    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 11:23:02.497697 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.497680    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 11:23:02.498124 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.498110    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 11:23:02.498859 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.498840    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-mxfqp\""
May 25 11:23:02.499073 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.499057    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 11:23:02.516137 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.516106    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"]
May 25 11:23:02.585643 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.585602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtflj\" (UniqueName: \"kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-kube-api-access-gtflj\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.585643 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.585648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.585862 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.585722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8bc4daa6-de1b-46cf-8fff-802c2774259d-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.686160 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.686130    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtflj\" (UniqueName: \"kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-kube-api-access-gtflj\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.686330 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.686168    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.686330 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.686215    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8bc4daa6-de1b-46cf-8fff-802c2774259d-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.686448 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:23:02.686358    2566 secret.go:281] references non-existent secret key: tls.crt
May 25 11:23:02.686448 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:23:02.686383    2566 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:23:02.686448 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:23:02.686404    2566 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
May 25 11:23:02.686448 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:23:02.686429    2566 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 25 11:23:02.686641 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:23:02.686509    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-certificates podName:8bc4daa6-de1b-46cf-8fff-802c2774259d nodeName:}" failed. No retries permitted until 2026-05-25 11:23:03.186487827 +0000 UTC m=+341.941464617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-certificates") pod "keda-metrics-apiserver-7c9f485588-xtmxt" (UID: "8bc4daa6-de1b-46cf-8fff-802c2774259d") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 25 11:23:02.686641 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.686604    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8bc4daa6-de1b-46cf-8fff-802c2774259d-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.699231 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.699208    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtflj\" (UniqueName: \"kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-kube-api-access-gtflj\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:02.836400 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.836326    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-8s8m4"]
May 25 11:23:02.839554 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.839502    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:02.843022 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.842998    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 25 11:23:02.859000 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.858971    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-8s8m4"]
May 25 11:23:02.989728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.989691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvbl\" (UniqueName: \"kubernetes.io/projected/b8ab9fc6-f6ba-455d-a361-8fc4cc449000-kube-api-access-bpvbl\") pod \"keda-admission-cf49989db-8s8m4\" (UID: \"b8ab9fc6-f6ba-455d-a361-8fc4cc449000\") " pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:02.989908 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:02.989747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b8ab9fc6-f6ba-455d-a361-8fc4cc449000-certificates\") pod \"keda-admission-cf49989db-8s8m4\" (UID: \"b8ab9fc6-f6ba-455d-a361-8fc4cc449000\") " pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:03.091055 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.091029    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b8ab9fc6-f6ba-455d-a361-8fc4cc449000-certificates\") pod \"keda-admission-cf49989db-8s8m4\" (UID: \"b8ab9fc6-f6ba-455d-a361-8fc4cc449000\") " pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:03.091211 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.091093    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvbl\" (UniqueName: \"kubernetes.io/projected/b8ab9fc6-f6ba-455d-a361-8fc4cc449000-kube-api-access-bpvbl\") pod \"keda-admission-cf49989db-8s8m4\" (UID: \"b8ab9fc6-f6ba-455d-a361-8fc4cc449000\") " pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:03.093439 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.093417    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b8ab9fc6-f6ba-455d-a361-8fc4cc449000-certificates\") pod \"keda-admission-cf49989db-8s8m4\" (UID: \"b8ab9fc6-f6ba-455d-a361-8fc4cc449000\") " pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:03.109206 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.109183    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvbl\" (UniqueName: \"kubernetes.io/projected/b8ab9fc6-f6ba-455d-a361-8fc4cc449000-kube-api-access-bpvbl\") pod \"keda-admission-cf49989db-8s8m4\" (UID: \"b8ab9fc6-f6ba-455d-a361-8fc4cc449000\") " pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:03.149756 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.149720    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:03.192075 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.192040    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:03.194480 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.194452    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8bc4daa6-de1b-46cf-8fff-802c2774259d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-xtmxt\" (UID: \"8bc4daa6-de1b-46cf-8fff-802c2774259d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:03.276046 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.276009    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-8s8m4"]
May 25 11:23:03.279267 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:23:03.279231    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ab9fc6_f6ba_455d_a361_8fc4cc449000.slice/crio-c1110ad2d486aa74bd92ae38a91ea4773646d0d0323802fff900bcd1894bbd05 WatchSource:0}: Error finding container c1110ad2d486aa74bd92ae38a91ea4773646d0d0323802fff900bcd1894bbd05: Status 404 returned error can't find the container with id c1110ad2d486aa74bd92ae38a91ea4773646d0d0323802fff900bcd1894bbd05
May 25 11:23:03.402209 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.402119    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:03.526675 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.526609    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"]
May 25 11:23:03.529074 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:23:03.529051    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc4daa6_de1b_46cf_8fff_802c2774259d.slice/crio-fe16afa05fea30ddaf52518f5fd65354d00357f53eca4912cb12b2ee972316a0 WatchSource:0}: Error finding container fe16afa05fea30ddaf52518f5fd65354d00357f53eca4912cb12b2ee972316a0: Status 404 returned error can't find the container with id fe16afa05fea30ddaf52518f5fd65354d00357f53eca4912cb12b2ee972316a0
May 25 11:23:03.862403 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.862362    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt" event={"ID":"8bc4daa6-de1b-46cf-8fff-802c2774259d","Type":"ContainerStarted","Data":"fe16afa05fea30ddaf52518f5fd65354d00357f53eca4912cb12b2ee972316a0"}
May 25 11:23:03.863667 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:03.863631    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-8s8m4" event={"ID":"b8ab9fc6-f6ba-455d-a361-8fc4cc449000","Type":"ContainerStarted","Data":"c1110ad2d486aa74bd92ae38a91ea4773646d0d0323802fff900bcd1894bbd05"}
May 25 11:23:04.868899 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:04.868863    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-8s8m4" event={"ID":"b8ab9fc6-f6ba-455d-a361-8fc4cc449000","Type":"ContainerStarted","Data":"4c1e339f373f6a8b34100f98d98c4a36242ee951b17747f901ca81d633ce39b7"}
May 25 11:23:04.869337 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:04.869015    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:04.890208 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:04.890143    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-8s8m4" podStartSLOduration=1.828372108 podStartE2EDuration="2.890123041s" podCreationTimestamp="2026-05-25 11:23:02 +0000 UTC" firstStartedPulling="2026-05-25 11:23:03.280489223 +0000 UTC m=+342.035466001" lastFinishedPulling="2026-05-25 11:23:04.342240149 +0000 UTC m=+343.097216934" observedRunningTime="2026-05-25 11:23:04.888200323 +0000 UTC m=+343.643177121" watchObservedRunningTime="2026-05-25 11:23:04.890123041 +0000 UTC m=+343.645099839"
May 25 11:23:06.875535 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:06.875500    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt" event={"ID":"8bc4daa6-de1b-46cf-8fff-802c2774259d","Type":"ContainerStarted","Data":"72e234f7f92284fddb85ed0b0bcb5db91c1a119bc040aaa1799aad70486b4dcf"}
May 25 11:23:06.899079 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:06.898996    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt" podStartSLOduration=2.171815019 podStartE2EDuration="4.898980162s" podCreationTimestamp="2026-05-25 11:23:02 +0000 UTC" firstStartedPulling="2026-05-25 11:23:03.530362394 +0000 UTC m=+342.285339172" lastFinishedPulling="2026-05-25 11:23:06.257527538 +0000 UTC m=+345.012504315" observedRunningTime="2026-05-25 11:23:06.898150726 +0000 UTC m=+345.653127520" watchObservedRunningTime="2026-05-25 11:23:06.898980162 +0000 UTC m=+345.653956960"
May 25 11:23:07.879170 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:07.879132    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:18.886173 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:18.886142    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-xtmxt"
May 25 11:23:25.874384 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:25.874347    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-8s8m4"
May 25 11:23:56.104890 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.104853    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"]
May 25 11:23:56.112751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.112727    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.115300 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.115274    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:23:56.115412 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.115391    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rvpn2\""
May 25 11:23:56.116280 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.116250    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:23:56.118018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.117998    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"]
May 25 11:23:56.163016 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.162984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.163016 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.163018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.163241 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.163037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9ck\" (UniqueName: \"kubernetes.io/projected/b210ccc6-9616-4531-aaea-9a19102037a2-kube-api-access-zq9ck\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.264327 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.264286    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.264327 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.264326    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.264633 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.264349    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9ck\" (UniqueName: \"kubernetes.io/projected/b210ccc6-9616-4531-aaea-9a19102037a2-kube-api-access-zq9ck\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.264789 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.264763    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.264848 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.264792    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.272772 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.272750    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9ck\" (UniqueName: \"kubernetes.io/projected/b210ccc6-9616-4531-aaea-9a19102037a2-kube-api-access-zq9ck\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.423353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.423272    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:23:56.545091 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:56.545062    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"]
May 25 11:23:56.548305 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:23:56.548277    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb210ccc6_9616_4531_aaea_9a19102037a2.slice/crio-048a1f6482753e2b39a7244cc4c004f92b1ab5053fc4d750f9b4f220a89b6bf2 WatchSource:0}: Error finding container 048a1f6482753e2b39a7244cc4c004f92b1ab5053fc4d750f9b4f220a89b6bf2: Status 404 returned error can't find the container with id 048a1f6482753e2b39a7244cc4c004f92b1ab5053fc4d750f9b4f220a89b6bf2
May 25 11:23:57.005845 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:57.005810    2566 generic.go:358] "Generic (PLEG): container finished" podID="b210ccc6-9616-4531-aaea-9a19102037a2" containerID="9e1effa31b900fe9f965c3eabad18cba3b2f9c1181cb60475c3e446b37eb7328" exitCode=0
May 25 11:23:57.006021 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:57.005898    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb" event={"ID":"b210ccc6-9616-4531-aaea-9a19102037a2","Type":"ContainerDied","Data":"9e1effa31b900fe9f965c3eabad18cba3b2f9c1181cb60475c3e446b37eb7328"}
May 25 11:23:57.006021 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:57.005936    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb" event={"ID":"b210ccc6-9616-4531-aaea-9a19102037a2","Type":"ContainerStarted","Data":"048a1f6482753e2b39a7244cc4c004f92b1ab5053fc4d750f9b4f220a89b6bf2"}
May 25 11:23:58.010935 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:58.010898    2566 generic.go:358] "Generic (PLEG): container finished" podID="b210ccc6-9616-4531-aaea-9a19102037a2" containerID="40b6dbf50985e2cca58176130dbd9e94dd78878bffbbebe2176e2074574b542e" exitCode=0
May 25 11:23:58.011422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:58.010950    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb" event={"ID":"b210ccc6-9616-4531-aaea-9a19102037a2","Type":"ContainerDied","Data":"40b6dbf50985e2cca58176130dbd9e94dd78878bffbbebe2176e2074574b542e"}
May 25 11:23:59.015154 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:59.015117    2566 generic.go:358] "Generic (PLEG): container finished" podID="b210ccc6-9616-4531-aaea-9a19102037a2" containerID="29f60e6929c2d89b8175b657abc1298d67edeb67b8af88526e1493f98bfe6b54" exitCode=0
May 25 11:23:59.015529 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:23:59.015170    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb" event={"ID":"b210ccc6-9616-4531-aaea-9a19102037a2","Type":"ContainerDied","Data":"29f60e6929c2d89b8175b657abc1298d67edeb67b8af88526e1493f98bfe6b54"}
May 25 11:24:00.138581 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.138559    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:24:00.191295 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.191260    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-bundle\") pod \"b210ccc6-9616-4531-aaea-9a19102037a2\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") "
May 25 11:24:00.191448 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.191327    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-util\") pod \"b210ccc6-9616-4531-aaea-9a19102037a2\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") "
May 25 11:24:00.191448 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.191362    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq9ck\" (UniqueName: \"kubernetes.io/projected/b210ccc6-9616-4531-aaea-9a19102037a2-kube-api-access-zq9ck\") pod \"b210ccc6-9616-4531-aaea-9a19102037a2\" (UID: \"b210ccc6-9616-4531-aaea-9a19102037a2\") "
May 25 11:24:00.191986 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.191959    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-bundle" (OuterVolumeSpecName: "bundle") pod "b210ccc6-9616-4531-aaea-9a19102037a2" (UID: "b210ccc6-9616-4531-aaea-9a19102037a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:24:00.193439 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.193405    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b210ccc6-9616-4531-aaea-9a19102037a2-kube-api-access-zq9ck" (OuterVolumeSpecName: "kube-api-access-zq9ck") pod "b210ccc6-9616-4531-aaea-9a19102037a2" (UID: "b210ccc6-9616-4531-aaea-9a19102037a2"). InnerVolumeSpecName "kube-api-access-zq9ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:24:00.196819 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.196797    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-util" (OuterVolumeSpecName: "util") pod "b210ccc6-9616-4531-aaea-9a19102037a2" (UID: "b210ccc6-9616-4531-aaea-9a19102037a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:24:00.292019 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.291926    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:00.292019 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.291969    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b210ccc6-9616-4531-aaea-9a19102037a2-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:00.292019 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:00.291979    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9ck\" (UniqueName: \"kubernetes.io/projected/b210ccc6-9616-4531-aaea-9a19102037a2-kube-api-access-zq9ck\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:01.023435 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:01.023398    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb" event={"ID":"b210ccc6-9616-4531-aaea-9a19102037a2","Type":"ContainerDied","Data":"048a1f6482753e2b39a7244cc4c004f92b1ab5053fc4d750f9b4f220a89b6bf2"}
May 25 11:24:01.023435 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:01.023432    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="048a1f6482753e2b39a7244cc4c004f92b1ab5053fc4d750f9b4f220a89b6bf2"
May 25 11:24:01.023656 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:01.023439    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19d5p6vb"
May 25 11:24:18.346257 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346223    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"]
May 25 11:24:18.346782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346567    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b210ccc6-9616-4531-aaea-9a19102037a2" containerName="util"
May 25 11:24:18.346782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346585    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b210ccc6-9616-4531-aaea-9a19102037a2" containerName="util"
May 25 11:24:18.346782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b210ccc6-9616-4531-aaea-9a19102037a2" containerName="pull"
May 25 11:24:18.346782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346624    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b210ccc6-9616-4531-aaea-9a19102037a2" containerName="pull"
May 25 11:24:18.346782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346637    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b210ccc6-9616-4531-aaea-9a19102037a2" containerName="extract"
May 25 11:24:18.346782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346647    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b210ccc6-9616-4531-aaea-9a19102037a2" containerName="extract"
May 25 11:24:18.346782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.346704    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="b210ccc6-9616-4531-aaea-9a19102037a2" containerName="extract"
May 25 11:24:18.349712 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.349693    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.353489 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.353471    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rvpn2\""
May 25 11:24:18.358934 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.358917    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:24:18.359011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.358975    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:24:18.369742 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.369723    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"]
May 25 11:24:18.423397 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.423359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8jx\" (UniqueName: \"kubernetes.io/projected/760ee1df-ba66-4741-887d-4b8ebfe6f278-kube-api-access-bk8jx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.423397 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.423397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.423637 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.423422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.524738 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.524706    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.524874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.524784    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8jx\" (UniqueName: \"kubernetes.io/projected/760ee1df-ba66-4741-887d-4b8ebfe6f278-kube-api-access-bk8jx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.524874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.524808    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.525150 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.525137    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.525192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.525132    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.536189 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.536157    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8jx\" (UniqueName: \"kubernetes.io/projected/760ee1df-ba66-4741-887d-4b8ebfe6f278-kube-api-access-bk8jx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.658322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.658222    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:18.783649 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:18.783617    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"]
May 25 11:24:18.786322 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:24:18.786294    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760ee1df_ba66_4741_887d_4b8ebfe6f278.slice/crio-3300f4894d13c09369130c3be547eddc75c54f0101f56f2443709906cbe46c3f WatchSource:0}: Error finding container 3300f4894d13c09369130c3be547eddc75c54f0101f56f2443709906cbe46c3f: Status 404 returned error can't find the container with id 3300f4894d13c09369130c3be547eddc75c54f0101f56f2443709906cbe46c3f
May 25 11:24:19.071439 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:19.071348    2566 generic.go:358] "Generic (PLEG): container finished" podID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerID="f0ec098eb08cd10bbc417b3344dbcdbd5e29d57e416836300e9b408731589022" exitCode=0
May 25 11:24:19.071439 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:19.071408    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x" event={"ID":"760ee1df-ba66-4741-887d-4b8ebfe6f278","Type":"ContainerDied","Data":"f0ec098eb08cd10bbc417b3344dbcdbd5e29d57e416836300e9b408731589022"}
May 25 11:24:19.071439 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:19.071437    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x" event={"ID":"760ee1df-ba66-4741-887d-4b8ebfe6f278","Type":"ContainerStarted","Data":"3300f4894d13c09369130c3be547eddc75c54f0101f56f2443709906cbe46c3f"}
May 25 11:24:22.082583 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:22.082534    2566 generic.go:358] "Generic (PLEG): container finished" podID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerID="d4251080f0de421bacb42512a909f5afb0b4682c30598c15189d526c18cb8921" exitCode=0
May 25 11:24:22.083006 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:22.082643    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x" event={"ID":"760ee1df-ba66-4741-887d-4b8ebfe6f278","Type":"ContainerDied","Data":"d4251080f0de421bacb42512a909f5afb0b4682c30598c15189d526c18cb8921"}
May 25 11:24:23.086693 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:23.086657    2566 generic.go:358] "Generic (PLEG): container finished" podID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerID="505e3cd882607be6c99551aecb859f7c8a7d4c664318961ba65090ada2a1c96f" exitCode=0
May 25 11:24:23.087065 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:23.086739    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x" event={"ID":"760ee1df-ba66-4741-887d-4b8ebfe6f278","Type":"ContainerDied","Data":"505e3cd882607be6c99551aecb859f7c8a7d4c664318961ba65090ada2a1c96f"}
May 25 11:24:24.207300 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.207279    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:24.269422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.269389    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-util\") pod \"760ee1df-ba66-4741-887d-4b8ebfe6f278\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") "
May 25 11:24:24.269422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.269428    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk8jx\" (UniqueName: \"kubernetes.io/projected/760ee1df-ba66-4741-887d-4b8ebfe6f278-kube-api-access-bk8jx\") pod \"760ee1df-ba66-4741-887d-4b8ebfe6f278\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") "
May 25 11:24:24.269680 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.269473    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-bundle\") pod \"760ee1df-ba66-4741-887d-4b8ebfe6f278\" (UID: \"760ee1df-ba66-4741-887d-4b8ebfe6f278\") "
May 25 11:24:24.269847 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.269826    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-bundle" (OuterVolumeSpecName: "bundle") pod "760ee1df-ba66-4741-887d-4b8ebfe6f278" (UID: "760ee1df-ba66-4741-887d-4b8ebfe6f278"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:24:24.271610 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.271517    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760ee1df-ba66-4741-887d-4b8ebfe6f278-kube-api-access-bk8jx" (OuterVolumeSpecName: "kube-api-access-bk8jx") pod "760ee1df-ba66-4741-887d-4b8ebfe6f278" (UID: "760ee1df-ba66-4741-887d-4b8ebfe6f278"). InnerVolumeSpecName "kube-api-access-bk8jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:24:24.274832 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.274814    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-util" (OuterVolumeSpecName: "util") pod "760ee1df-ba66-4741-887d-4b8ebfe6f278" (UID: "760ee1df-ba66-4741-887d-4b8ebfe6f278"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:24:24.371064 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.370981    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:24.371064 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.371014    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/760ee1df-ba66-4741-887d-4b8ebfe6f278-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:24.371064 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:24.371023    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk8jx\" (UniqueName: \"kubernetes.io/projected/760ee1df-ba66-4741-887d-4b8ebfe6f278-kube-api-access-bk8jx\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:25.093287 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:25.093251    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x" event={"ID":"760ee1df-ba66-4741-887d-4b8ebfe6f278","Type":"ContainerDied","Data":"3300f4894d13c09369130c3be547eddc75c54f0101f56f2443709906cbe46c3f"}
May 25 11:24:25.093287 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:25.093291    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3300f4894d13c09369130c3be547eddc75c54f0101f56f2443709906cbe46c3f"
May 25 11:24:25.093287 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:25.093297    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft6q9x"
May 25 11:24:49.034621 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.034530    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"]
May 25 11:24:49.035159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.034882    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerName="extract"
May 25 11:24:49.035159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.034899    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerName="extract"
May 25 11:24:49.035159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.034916    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerName="pull"
May 25 11:24:49.035159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.034925    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerName="pull"
May 25 11:24:49.035159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.034940    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerName="util"
May 25 11:24:49.035159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.034948    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerName="util"
May 25 11:24:49.035159 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.035012    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="760ee1df-ba66-4741-887d-4b8ebfe6f278" containerName="extract"
May 25 11:24:49.042478 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.042454    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.045036 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.045007    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:24:49.045172 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.045075    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:24:49.045358 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.045338    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rvpn2\""
May 25 11:24:49.045657 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.045638    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"]
May 25 11:24:49.142886 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.142846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w945v\" (UniqueName: \"kubernetes.io/projected/88392ad4-0aa3-466a-9336-4db2e6f5b128-kube-api-access-w945v\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.142886 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.142891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.143147 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.143025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.243989 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.243936    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.243989 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.244002    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w945v\" (UniqueName: \"kubernetes.io/projected/88392ad4-0aa3-466a-9336-4db2e6f5b128-kube-api-access-w945v\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.244189 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.244029    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.244336 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.244316    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.244380 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.244343    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.252786 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.252755    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w945v\" (UniqueName: \"kubernetes.io/projected/88392ad4-0aa3-466a-9336-4db2e6f5b128-kube-api-access-w945v\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.351946 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.351909    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:49.478074 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:49.478037    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"]
May 25 11:24:49.482244 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:24:49.482219    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88392ad4_0aa3_466a_9336_4db2e6f5b128.slice/crio-04d2cce201c3462ad05662078fb34efbc5f964d9014ac29f3d1fc5e5926c4068 WatchSource:0}: Error finding container 04d2cce201c3462ad05662078fb34efbc5f964d9014ac29f3d1fc5e5926c4068: Status 404 returned error can't find the container with id 04d2cce201c3462ad05662078fb34efbc5f964d9014ac29f3d1fc5e5926c4068
May 25 11:24:50.162554 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:50.162511    2566 generic.go:358] "Generic (PLEG): container finished" podID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerID="746e4392254b6900ed09f0c2127667c50d143a523ca9fd53c6451f78eb19090f" exitCode=0
May 25 11:24:50.162918 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:50.162592    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c" event={"ID":"88392ad4-0aa3-466a-9336-4db2e6f5b128","Type":"ContainerDied","Data":"746e4392254b6900ed09f0c2127667c50d143a523ca9fd53c6451f78eb19090f"}
May 25 11:24:50.162918 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:50.162625    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c" event={"ID":"88392ad4-0aa3-466a-9336-4db2e6f5b128","Type":"ContainerStarted","Data":"04d2cce201c3462ad05662078fb34efbc5f964d9014ac29f3d1fc5e5926c4068"}
May 25 11:24:51.167125 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:51.167092    2566 generic.go:358] "Generic (PLEG): container finished" podID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerID="79c5d9822de18e068ca9fa1ff4f7a19bd525004a34be2fd95886f6bd47d92c28" exitCode=0
May 25 11:24:51.167511 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:51.167132    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c" event={"ID":"88392ad4-0aa3-466a-9336-4db2e6f5b128","Type":"ContainerDied","Data":"79c5d9822de18e068ca9fa1ff4f7a19bd525004a34be2fd95886f6bd47d92c28"}
May 25 11:24:52.171611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:52.171569    2566 generic.go:358] "Generic (PLEG): container finished" podID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerID="02ab5854ad53371c9fba621d8067f42b984c4fd3df93f2724b691554fa999d43" exitCode=0
May 25 11:24:52.171998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:52.171643    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c" event={"ID":"88392ad4-0aa3-466a-9336-4db2e6f5b128","Type":"ContainerDied","Data":"02ab5854ad53371c9fba621d8067f42b984c4fd3df93f2724b691554fa999d43"}
May 25 11:24:53.289846 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.289821    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:53.378166 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.378126    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-util\") pod \"88392ad4-0aa3-466a-9336-4db2e6f5b128\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") "
May 25 11:24:53.378166 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.378174    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-bundle\") pod \"88392ad4-0aa3-466a-9336-4db2e6f5b128\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") "
May 25 11:24:53.378417 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.378205    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w945v\" (UniqueName: \"kubernetes.io/projected/88392ad4-0aa3-466a-9336-4db2e6f5b128-kube-api-access-w945v\") pod \"88392ad4-0aa3-466a-9336-4db2e6f5b128\" (UID: \"88392ad4-0aa3-466a-9336-4db2e6f5b128\") "
May 25 11:24:53.379075 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.379048    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-bundle" (OuterVolumeSpecName: "bundle") pod "88392ad4-0aa3-466a-9336-4db2e6f5b128" (UID: "88392ad4-0aa3-466a-9336-4db2e6f5b128"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:24:53.380254 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.380229    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88392ad4-0aa3-466a-9336-4db2e6f5b128-kube-api-access-w945v" (OuterVolumeSpecName: "kube-api-access-w945v") pod "88392ad4-0aa3-466a-9336-4db2e6f5b128" (UID: "88392ad4-0aa3-466a-9336-4db2e6f5b128"). InnerVolumeSpecName "kube-api-access-w945v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:24:53.384141 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.384120    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-util" (OuterVolumeSpecName: "util") pod "88392ad4-0aa3-466a-9336-4db2e6f5b128" (UID: "88392ad4-0aa3-466a-9336-4db2e6f5b128"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:24:53.478723 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.478638    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:53.478723 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.478668    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88392ad4-0aa3-466a-9336-4db2e6f5b128-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:53.478723 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:53.478677    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w945v\" (UniqueName: \"kubernetes.io/projected/88392ad4-0aa3-466a-9336-4db2e6f5b128-kube-api-access-w945v\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:24:54.178399 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:54.178314    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c" event={"ID":"88392ad4-0aa3-466a-9336-4db2e6f5b128","Type":"ContainerDied","Data":"04d2cce201c3462ad05662078fb34efbc5f964d9014ac29f3d1fc5e5926c4068"}
May 25 11:24:54.178399 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:54.178350    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d2cce201c3462ad05662078fb34efbc5f964d9014ac29f3d1fc5e5926c4068"
May 25 11:24:54.178676 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:54.178399    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835rk56c"
May 25 11:24:58.947501 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947471    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"]
May 25 11:24:58.947884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947757    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerName="extract"
May 25 11:24:58.947884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947769    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerName="extract"
May 25 11:24:58.947884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947779    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerName="util"
May 25 11:24:58.947884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947787    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerName="util"
May 25 11:24:58.947884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947796    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerName="pull"
May 25 11:24:58.947884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947804    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerName="pull"
May 25 11:24:58.947884 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.947843    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="88392ad4-0aa3-466a-9336-4db2e6f5b128" containerName="extract"
May 25 11:24:58.951940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.951923    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:58.954583 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.954563    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rvpn2\""
May 25 11:24:58.954878 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.954862    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:24:58.955305 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.955288    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:24:58.969413 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:58.969388    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"]
May 25 11:24:59.119520 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.119476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n4l\" (UniqueName: \"kubernetes.io/projected/69cb23fd-6ab6-4bdc-a621-2b162a384099-kube-api-access-v2n4l\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.119520 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.119516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.119758 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.119579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.220868 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.220761    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n4l\" (UniqueName: \"kubernetes.io/projected/69cb23fd-6ab6-4bdc-a621-2b162a384099-kube-api-access-v2n4l\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.220868 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.220816    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.220868 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.220849    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.221176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.221161    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.221254 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.221235    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.236738 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.236710    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n4l\" (UniqueName: \"kubernetes.io/projected/69cb23fd-6ab6-4bdc-a621-2b162a384099-kube-api-access-v2n4l\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.260568 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.260528    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:24:59.398978 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.398943    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"]
May 25 11:24:59.402261 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:24:59.402235    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cb23fd_6ab6_4bdc_a621_2b162a384099.slice/crio-0c242bc649918ae6424a58d4d0e4039d22fbbbbd90f4ef9fee48c0be54b5c09a WatchSource:0}: Error finding container 0c242bc649918ae6424a58d4d0e4039d22fbbbbd90f4ef9fee48c0be54b5c09a: Status 404 returned error can't find the container with id 0c242bc649918ae6424a58d4d0e4039d22fbbbbd90f4ef9fee48c0be54b5c09a
May 25 11:24:59.869228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.869194    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"]
May 25 11:24:59.872572 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.872534    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:24:59.877611 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.877591    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\""
May 25 11:24:59.877724 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.877626    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\""
May 25 11:24:59.879638 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.879619    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-wnwvg\""
May 25 11:24:59.937554 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:24:59.937511    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"]
May 25 11:25:00.026044 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.026015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/83257f6b-8f56-4a1a-a016-4b9874c08a9e-operator-config\") pod \"servicemesh-operator3-55f49c5f94-q29qs\" (UID: \"83257f6b-8f56-4a1a-a016-4b9874c08a9e\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:00.026044 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.026048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmmnv\" (UniqueName: \"kubernetes.io/projected/83257f6b-8f56-4a1a-a016-4b9874c08a9e-kube-api-access-kmmnv\") pod \"servicemesh-operator3-55f49c5f94-q29qs\" (UID: \"83257f6b-8f56-4a1a-a016-4b9874c08a9e\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:00.127346 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.127258    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/83257f6b-8f56-4a1a-a016-4b9874c08a9e-operator-config\") pod \"servicemesh-operator3-55f49c5f94-q29qs\" (UID: \"83257f6b-8f56-4a1a-a016-4b9874c08a9e\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:00.127346 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.127300    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmnv\" (UniqueName: \"kubernetes.io/projected/83257f6b-8f56-4a1a-a016-4b9874c08a9e-kube-api-access-kmmnv\") pod \"servicemesh-operator3-55f49c5f94-q29qs\" (UID: \"83257f6b-8f56-4a1a-a016-4b9874c08a9e\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:00.129782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.129765    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/83257f6b-8f56-4a1a-a016-4b9874c08a9e-operator-config\") pod \"servicemesh-operator3-55f49c5f94-q29qs\" (UID: \"83257f6b-8f56-4a1a-a016-4b9874c08a9e\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:00.143910 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.143890    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmnv\" (UniqueName: \"kubernetes.io/projected/83257f6b-8f56-4a1a-a016-4b9874c08a9e-kube-api-access-kmmnv\") pod \"servicemesh-operator3-55f49c5f94-q29qs\" (UID: \"83257f6b-8f56-4a1a-a016-4b9874c08a9e\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:00.180900 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.180869    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:00.197465 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.197434    2566 generic.go:358] "Generic (PLEG): container finished" podID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerID="248cab3c2fff302f81420834cc2eb303ab08418d5c59aaf9ae14b6f440f75d8e" exitCode=0
May 25 11:25:00.197640 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.197513    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn" event={"ID":"69cb23fd-6ab6-4bdc-a621-2b162a384099","Type":"ContainerDied","Data":"248cab3c2fff302f81420834cc2eb303ab08418d5c59aaf9ae14b6f440f75d8e"}
May 25 11:25:00.197640 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.197558    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn" event={"ID":"69cb23fd-6ab6-4bdc-a621-2b162a384099","Type":"ContainerStarted","Data":"0c242bc649918ae6424a58d4d0e4039d22fbbbbd90f4ef9fee48c0be54b5c09a"}
May 25 11:25:00.331694 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:00.331661    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"]
May 25 11:25:00.335314 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:00.335273    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83257f6b_8f56_4a1a_a016_4b9874c08a9e.slice/crio-3677de5104fb55acc66122697217c32873b807c5621b2a2e35f93133b8836418 WatchSource:0}: Error finding container 3677de5104fb55acc66122697217c32873b807c5621b2a2e35f93133b8836418: Status 404 returned error can't find the container with id 3677de5104fb55acc66122697217c32873b807c5621b2a2e35f93133b8836418
May 25 11:25:01.201809 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:01.201772    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs" event={"ID":"83257f6b-8f56-4a1a-a016-4b9874c08a9e","Type":"ContainerStarted","Data":"3677de5104fb55acc66122697217c32873b807c5621b2a2e35f93133b8836418"}
May 25 11:25:01.204303 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:01.204270    2566 generic.go:358] "Generic (PLEG): container finished" podID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerID="af63393e8d5d46a9049935f573d07251b74f99f9cf89a1ca5e27d827f63bfd4e" exitCode=0
May 25 11:25:01.204445 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:01.204314    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn" event={"ID":"69cb23fd-6ab6-4bdc-a621-2b162a384099","Type":"ContainerDied","Data":"af63393e8d5d46a9049935f573d07251b74f99f9cf89a1ca5e27d827f63bfd4e"}
May 25 11:25:02.210328 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:02.210289    2566 generic.go:358] "Generic (PLEG): container finished" podID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerID="2bbb72e069eab98bb1066ba6b301fbe67a568a02cb69197f844f46ba0985980f" exitCode=0
May 25 11:25:02.210767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:02.210354    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn" event={"ID":"69cb23fd-6ab6-4bdc-a621-2b162a384099","Type":"ContainerDied","Data":"2bbb72e069eab98bb1066ba6b301fbe67a568a02cb69197f844f46ba0985980f"}
May 25 11:25:03.339018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.338986    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:25:03.456568 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.456523    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-util\") pod \"69cb23fd-6ab6-4bdc-a621-2b162a384099\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") "
May 25 11:25:03.456716 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.456583    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2n4l\" (UniqueName: \"kubernetes.io/projected/69cb23fd-6ab6-4bdc-a621-2b162a384099-kube-api-access-v2n4l\") pod \"69cb23fd-6ab6-4bdc-a621-2b162a384099\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") "
May 25 11:25:03.456716 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.456602    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-bundle\") pod \"69cb23fd-6ab6-4bdc-a621-2b162a384099\" (UID: \"69cb23fd-6ab6-4bdc-a621-2b162a384099\") "
May 25 11:25:03.457464 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.457437    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-bundle" (OuterVolumeSpecName: "bundle") pod "69cb23fd-6ab6-4bdc-a621-2b162a384099" (UID: "69cb23fd-6ab6-4bdc-a621-2b162a384099"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:03.458465 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.458442    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cb23fd-6ab6-4bdc-a621-2b162a384099-kube-api-access-v2n4l" (OuterVolumeSpecName: "kube-api-access-v2n4l") pod "69cb23fd-6ab6-4bdc-a621-2b162a384099" (UID: "69cb23fd-6ab6-4bdc-a621-2b162a384099"). InnerVolumeSpecName "kube-api-access-v2n4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:03.464461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.464415    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-util" (OuterVolumeSpecName: "util") pod "69cb23fd-6ab6-4bdc-a621-2b162a384099" (UID: "69cb23fd-6ab6-4bdc-a621-2b162a384099"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:03.557785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.557694    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:03.557785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.557725    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2n4l\" (UniqueName: \"kubernetes.io/projected/69cb23fd-6ab6-4bdc-a621-2b162a384099-kube-api-access-v2n4l\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:03.557785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:03.557738    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69cb23fd-6ab6-4bdc-a621-2b162a384099-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:04.218885 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:04.218790    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn" event={"ID":"69cb23fd-6ab6-4bdc-a621-2b162a384099","Type":"ContainerDied","Data":"0c242bc649918ae6424a58d4d0e4039d22fbbbbd90f4ef9fee48c0be54b5c09a"}
May 25 11:25:04.218885 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:04.218832    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c242bc649918ae6424a58d4d0e4039d22fbbbbd90f4ef9fee48c0be54b5c09a"
May 25 11:25:04.218885 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:04.218802    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762a42lxn"
May 25 11:25:04.220422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:04.220396    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs" event={"ID":"83257f6b-8f56-4a1a-a016-4b9874c08a9e","Type":"ContainerStarted","Data":"af1e026b99951d9c82660127671f3fcc3a6142de29ec6406adc16195c1f75cf1"}
May 25 11:25:04.220560 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:04.220489    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:04.241633 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:04.241588    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs" podStartSLOduration=2.204792764 podStartE2EDuration="5.241571835s" podCreationTimestamp="2026-05-25 11:24:59 +0000 UTC" firstStartedPulling="2026-05-25 11:25:00.337740539 +0000 UTC m=+459.092717316" lastFinishedPulling="2026-05-25 11:25:03.37451961 +0000 UTC m=+462.129496387" observedRunningTime="2026-05-25 11:25:04.23933184 +0000 UTC m=+462.994308637" watchObservedRunningTime="2026-05-25 11:25:04.241571835 +0000 UTC m=+462.996548631"
May 25 11:25:13.362980 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.362944    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"]
May 25 11:25:13.363422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.363236    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerName="util"
May 25 11:25:13.363422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.363249    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerName="util"
May 25 11:25:13.363422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.363261    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerName="pull"
May 25 11:25:13.363422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.363266    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerName="pull"
May 25 11:25:13.363422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.363274    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerName="extract"
May 25 11:25:13.363422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.363282    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerName="extract"
May 25 11:25:13.363422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.363334    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="69cb23fd-6ab6-4bdc-a621-2b162a384099" containerName="extract"
May 25 11:25:13.366185 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.366167    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.368722 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.368694    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
May 25 11:25:13.368722 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.368701    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
May 25 11:25:13.368909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.368783    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 25 11:25:13.368978 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.368964    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
May 25 11:25:13.369206 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.369191    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m7bds\""
May 25 11:25:13.369252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.369220    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 25 11:25:13.369252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.369228    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
May 25 11:25:13.377308 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.377287    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"]
May 25 11:25:13.427993 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.427958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.428182 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.428000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.428182 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.428020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.428182 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.428038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.428182 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.428109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f5541-a139-48c7-95b3-8aba4b627ca5-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.428182 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.428150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.428358 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.428222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7bmx\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-kube-api-access-g7bmx\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.529439 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.529404    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.529439 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.529440    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.529711 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.529460    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.529711 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.529589    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f5541-a139-48c7-95b3-8aba4b627ca5-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.529711 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.529625    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.529711 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.529679    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g7bmx\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-kube-api-access-g7bmx\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.529909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.529728    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.530294 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.530249    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.531925 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.531898    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f5541-a139-48c7-95b3-8aba4b627ca5-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.532328 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.532306    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.532401 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.532318    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.532457 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.532400    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.537269 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.537247    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.537350 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.537287    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7bmx\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-kube-api-access-g7bmx\") pod \"istiod-openshift-gateway-767cf5cf55-g9wbl\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.675359 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.675253    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:13.816923 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:13.816824    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"]
May 25 11:25:13.819592 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:13.819563    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f5541_a139_48c7_95b3_8aba4b627ca5.slice/crio-21c32ab6006eae425d9a825b42b5beb6f0a487fd835cfd774a7141af1d775786 WatchSource:0}: Error finding container 21c32ab6006eae425d9a825b42b5beb6f0a487fd835cfd774a7141af1d775786: Status 404 returned error can't find the container with id 21c32ab6006eae425d9a825b42b5beb6f0a487fd835cfd774a7141af1d775786
May 25 11:25:14.252586 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:14.252528    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl" event={"ID":"405f5541-a139-48c7-95b3-8aba4b627ca5","Type":"ContainerStarted","Data":"21c32ab6006eae425d9a825b42b5beb6f0a487fd835cfd774a7141af1d775786"}
May 25 11:25:15.226192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:15.226161    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-q29qs"
May 25 11:25:16.260573 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:16.260520    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:25:16.260840 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:16.260619    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:25:17.264908 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:17.264870    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl" event={"ID":"405f5541-a139-48c7-95b3-8aba4b627ca5","Type":"ContainerStarted","Data":"245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa"}
May 25 11:25:17.265346 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:17.265090    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:17.266261 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:17.266233    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-767cf5cf55-g9wbl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
May 25 11:25:17.266373 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:17.266294    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl" podUID="405f5541-a139-48c7-95b3-8aba4b627ca5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:17.311257 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:17.311194    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl" podStartSLOduration=1.872337267 podStartE2EDuration="4.311169633s" podCreationTimestamp="2026-05-25 11:25:13 +0000 UTC" firstStartedPulling="2026-05-25 11:25:13.821499822 +0000 UTC m=+472.576476596" lastFinishedPulling="2026-05-25 11:25:16.260332189 +0000 UTC m=+475.015308962" observedRunningTime="2026-05-25 11:25:17.30843273 +0000 UTC m=+476.063409538" watchObservedRunningTime="2026-05-25 11:25:17.311169633 +0000 UTC m=+476.066146430"
May 25 11:25:18.268664 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:18.268639    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:25:20.255173 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.255141    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"]
May 25 11:25:20.258406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.258387    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.261059 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.261022    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-hh92m\""
May 25 11:25:20.270035 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.270004    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"]
May 25 11:25:20.286146 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286287 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzptc\" (UniqueName: \"kubernetes.io/projected/12d991fe-1987-486c-a052-7702a46b9555-kube-api-access-fzptc\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286287 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286367 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d991fe-1987-486c-a052-7702a46b9555-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286367 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286441 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d991fe-1987-486c-a052-7702a46b9555-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286478 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286515 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d991fe-1987-486c-a052-7702a46b9555-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.286567 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.286555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387270 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387233    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d991fe-1987-486c-a052-7702a46b9555-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387270 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387274    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387303    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387323    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fzptc\" (UniqueName: \"kubernetes.io/projected/12d991fe-1987-486c-a052-7702a46b9555-kube-api-access-fzptc\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387345    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387370    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d991fe-1987-486c-a052-7702a46b9555-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387414    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387452    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d991fe-1987-486c-a052-7702a46b9555-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.387507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387477    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.388033 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387931    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.388033 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.387967    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.388098 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.388028    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.388098 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.388062    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.388289 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.388271    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d991fe-1987-486c-a052-7702a46b9555-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.389684 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.389665    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d991fe-1987-486c-a052-7702a46b9555-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.389926 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.389909    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d991fe-1987-486c-a052-7702a46b9555-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.396336 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.396312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzptc\" (UniqueName: \"kubernetes.io/projected/12d991fe-1987-486c-a052-7702a46b9555-kube-api-access-fzptc\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.396589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.396565    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d991fe-1987-486c-a052-7702a46b9555-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8\" (UID: \"12d991fe-1987-486c-a052-7702a46b9555\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.571334 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.571222    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:20.715197 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:20.715120    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"]
May 25 11:25:20.722103 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:20.722074    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d991fe_1987_486c_a052_7702a46b9555.slice/crio-01eccc3ea3e76f4b93682eb21967acfde40b7966eb9f8b0e5f8df93b0f332ce9 WatchSource:0}: Error finding container 01eccc3ea3e76f4b93682eb21967acfde40b7966eb9f8b0e5f8df93b0f332ce9: Status 404 returned error can't find the container with id 01eccc3ea3e76f4b93682eb21967acfde40b7966eb9f8b0e5f8df93b0f332ce9
May 25 11:25:21.278297 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:21.278261    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8" event={"ID":"12d991fe-1987-486c-a052-7702a46b9555","Type":"ContainerStarted","Data":"01eccc3ea3e76f4b93682eb21967acfde40b7966eb9f8b0e5f8df93b0f332ce9"}
May 25 11:25:23.050191 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:23.050151    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:25:23.050583 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:23.050259    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:25:23.050583 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:23.050311    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:25:23.289252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:23.289216    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8" event={"ID":"12d991fe-1987-486c-a052-7702a46b9555","Type":"ContainerStarted","Data":"121622ea4b63b23f5927ae4dc77eab2c6c17639a95a57f868380c0313b680349"}
May 25 11:25:23.317300 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:23.317190    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8" podStartSLOduration=0.991879665 podStartE2EDuration="3.317170135s" podCreationTimestamp="2026-05-25 11:25:20 +0000 UTC" firstStartedPulling="2026-05-25 11:25:20.724640047 +0000 UTC m=+479.479616822" lastFinishedPulling="2026-05-25 11:25:23.049930515 +0000 UTC m=+481.804907292" observedRunningTime="2026-05-25 11:25:23.312902665 +0000 UTC m=+482.067879459" watchObservedRunningTime="2026-05-25 11:25:23.317170135 +0000 UTC m=+482.072146933"
May 25 11:25:23.571766 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:23.571672    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:24.576322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:24.576287    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:25.296225 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:25.296194    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:25.297119 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:25.297098    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8"
May 25 11:25:28.755876 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.755845    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"]
May 25 11:25:28.760372 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.760350    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.763520 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.763493    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:25:28.763760 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.763566    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:25:28.764434 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.764419    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rvpn2\""
May 25 11:25:28.773396 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.773369    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"]
May 25 11:25:28.859743 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.859711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pxf\" (UniqueName: \"kubernetes.io/projected/c6148c57-736c-48a4-8dfc-d67827349aa1-kube-api-access-d8pxf\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.859931 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.859761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.859931 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.859859    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.866616 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.866586    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"]
May 25 11:25:28.869868 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.869853    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:28.885172 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.885144    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"]
May 25 11:25:28.960365 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:28.960589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptph\" (UniqueName: \"kubernetes.io/projected/d5d5e079-8916-41de-befb-ccef9687add1-kube-api-access-hptph\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:28.960589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:28.960589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960473    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.960589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960526    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pxf\" (UniqueName: \"kubernetes.io/projected/c6148c57-736c-48a4-8dfc-d67827349aa1-kube-api-access-d8pxf\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.960589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960576    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.960921 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960900    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.960981 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.960935    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.970450 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.970419    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pxf\" (UniqueName: \"kubernetes.io/projected/c6148c57-736c-48a4-8dfc-d67827349aa1-kube-api-access-d8pxf\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:28.984275 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.984238    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"]
May 25 11:25:28.990071 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.990042    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:28.998880 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:28.998856    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"]
May 25 11:25:29.061045 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.060947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7stvl\" (UniqueName: \"kubernetes.io/projected/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-kube-api-access-7stvl\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.061045 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.061002    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:29.061045 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.061023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.061303 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.061133    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hptph\" (UniqueName: \"kubernetes.io/projected/d5d5e079-8916-41de-befb-ccef9687add1-kube-api-access-hptph\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:29.061303 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.061194    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:29.061303 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.061241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.061471 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.061412    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:29.061904 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.061875    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:29.063682 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.063658    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"]
May 25 11:25:29.067066 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.067047    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.069182 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.069164    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:29.076890 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.076844    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"]
May 25 11:25:29.078751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.078726    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptph\" (UniqueName: \"kubernetes.io/projected/d5d5e079-8916-41de-befb-ccef9687add1-kube-api-access-hptph\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:29.162144 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162107    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.162301 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.162301 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162238    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7stvl\" (UniqueName: \"kubernetes.io/projected/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-kube-api-access-7stvl\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.162301 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.162406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162311    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.162406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww45h\" (UniqueName: \"kubernetes.io/projected/76914f61-7410-46e4-a9d5-45fc74fb752b-kube-api-access-ww45h\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.162467 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162454    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.162750 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.162730    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.171637 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.171597    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7stvl\" (UniqueName: \"kubernetes.io/projected/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-kube-api-access-7stvl\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.178177 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.178149    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:29.201169 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.201140    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"]
May 25 11:25:29.203702 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:29.203674    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6148c57_736c_48a4_8dfc_d67827349aa1.slice/crio-ec96a8b6fd2c540e18861a617a5a4e7f8c3579da5d29f89881b9e7a0f9a1c7a8 WatchSource:0}: Error finding container ec96a8b6fd2c540e18861a617a5a4e7f8c3579da5d29f89881b9e7a0f9a1c7a8: Status 404 returned error can't find the container with id ec96a8b6fd2c540e18861a617a5a4e7f8c3579da5d29f89881b9e7a0f9a1c7a8
May 25 11:25:29.263192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.263145    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.263463 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.263213    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.263463 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.263273    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ww45h\" (UniqueName: \"kubernetes.io/projected/76914f61-7410-46e4-a9d5-45fc74fb752b-kube-api-access-ww45h\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.263630 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.263526    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.263630 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.263610    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.277474 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.277450    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww45h\" (UniqueName: \"kubernetes.io/projected/76914f61-7410-46e4-a9d5-45fc74fb752b-kube-api-access-ww45h\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.300731 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.300698    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:29.309568 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.309516    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56" event={"ID":"c6148c57-736c-48a4-8dfc-d67827349aa1","Type":"ContainerStarted","Data":"16473ee11c46ec9f206fde20b75ed109865526f087ddf44896707460931d841e"}
May 25 11:25:29.309678 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.309579    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56" event={"ID":"c6148c57-736c-48a4-8dfc-d67827349aa1","Type":"ContainerStarted","Data":"ec96a8b6fd2c540e18861a617a5a4e7f8c3579da5d29f89881b9e7a0f9a1c7a8"}
May 25 11:25:29.313992 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.313868    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"]
May 25 11:25:29.316110 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:29.316086    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d5e079_8916_41de_befb_ccef9687add1.slice/crio-a27bf0dd44fae4c5b0a3cc09cd97f29f8d617019f3af71165baf7516469f3185 WatchSource:0}: Error finding container a27bf0dd44fae4c5b0a3cc09cd97f29f8d617019f3af71165baf7516469f3185: Status 404 returned error can't find the container with id a27bf0dd44fae4c5b0a3cc09cd97f29f8d617019f3af71165baf7516469f3185
May 25 11:25:29.376973 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.376950    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:29.444408 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.444379    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"]
May 25 11:25:29.488856 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:29.488824    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa3fc5a_62e9_47fe_82c9_71444b9c7f43.slice/crio-243e1206347784d2efe7d5dcaf7af7910564da4a988ebc92ff952515315cf4cc WatchSource:0}: Error finding container 243e1206347784d2efe7d5dcaf7af7910564da4a988ebc92ff952515315cf4cc: Status 404 returned error can't find the container with id 243e1206347784d2efe7d5dcaf7af7910564da4a988ebc92ff952515315cf4cc
May 25 11:25:29.515811 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:29.515783    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"]
May 25 11:25:29.519717 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:29.519680    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76914f61_7410_46e4_a9d5_45fc74fb752b.slice/crio-0e614b6dd189ca0b71b13bf28727d6b688fb60bc6ada21977ec5d532fce1dda7 WatchSource:0}: Error finding container 0e614b6dd189ca0b71b13bf28727d6b688fb60bc6ada21977ec5d532fce1dda7: Status 404 returned error can't find the container with id 0e614b6dd189ca0b71b13bf28727d6b688fb60bc6ada21977ec5d532fce1dda7
May 25 11:25:30.313825 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.313788    2566 generic.go:358] "Generic (PLEG): container finished" podID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerID="16473ee11c46ec9f206fde20b75ed109865526f087ddf44896707460931d841e" exitCode=0
May 25 11:25:30.314251 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.313869    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56" event={"ID":"c6148c57-736c-48a4-8dfc-d67827349aa1","Type":"ContainerDied","Data":"16473ee11c46ec9f206fde20b75ed109865526f087ddf44896707460931d841e"}
May 25 11:25:30.315244 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.315197    2566 generic.go:358] "Generic (PLEG): container finished" podID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerID="2cf6c5bb75ab75338efafed9f39225eb99f254ce929175cc13e29c4534c2cc98" exitCode=0
May 25 11:25:30.315296 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.315256    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg" event={"ID":"afa3fc5a-62e9-47fe-82c9-71444b9c7f43","Type":"ContainerDied","Data":"2cf6c5bb75ab75338efafed9f39225eb99f254ce929175cc13e29c4534c2cc98"}
May 25 11:25:30.315296 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.315288    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg" event={"ID":"afa3fc5a-62e9-47fe-82c9-71444b9c7f43","Type":"ContainerStarted","Data":"243e1206347784d2efe7d5dcaf7af7910564da4a988ebc92ff952515315cf4cc"}
May 25 11:25:30.316775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.316702    2566 generic.go:358] "Generic (PLEG): container finished" podID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerID="5f6a2f60a7a871468a9493cfba694dd18d25aba2734d98c8a4d00c6854e01927" exitCode=0
May 25 11:25:30.316775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.316767    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp" event={"ID":"76914f61-7410-46e4-a9d5-45fc74fb752b","Type":"ContainerDied","Data":"5f6a2f60a7a871468a9493cfba694dd18d25aba2734d98c8a4d00c6854e01927"}
May 25 11:25:30.317018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.316785    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp" event={"ID":"76914f61-7410-46e4-a9d5-45fc74fb752b","Type":"ContainerStarted","Data":"0e614b6dd189ca0b71b13bf28727d6b688fb60bc6ada21977ec5d532fce1dda7"}
May 25 11:25:30.318186 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.318068    2566 generic.go:358] "Generic (PLEG): container finished" podID="d5d5e079-8916-41de-befb-ccef9687add1" containerID="a6ba09c543e6d4c0b26ecdcf8791c90d44368fb6822b61d385e160d6108396de" exitCode=0
May 25 11:25:30.318186 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.318149    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq" event={"ID":"d5d5e079-8916-41de-befb-ccef9687add1","Type":"ContainerDied","Data":"a6ba09c543e6d4c0b26ecdcf8791c90d44368fb6822b61d385e160d6108396de"}
May 25 11:25:30.318186 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:30.318181    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq" event={"ID":"d5d5e079-8916-41de-befb-ccef9687add1","Type":"ContainerStarted","Data":"a27bf0dd44fae4c5b0a3cc09cd97f29f8d617019f3af71165baf7516469f3185"}
May 25 11:25:31.322891 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:31.322806    2566 generic.go:358] "Generic (PLEG): container finished" podID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerID="1ef5b2697a26e6f8d18bb62ba0fe90657e7054bbc01258907ca85de02e72fd89" exitCode=0
May 25 11:25:31.323283 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:31.322888    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg" event={"ID":"afa3fc5a-62e9-47fe-82c9-71444b9c7f43","Type":"ContainerDied","Data":"1ef5b2697a26e6f8d18bb62ba0fe90657e7054bbc01258907ca85de02e72fd89"}
May 25 11:25:32.327959 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.327868    2566 generic.go:358] "Generic (PLEG): container finished" podID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerID="2723d096b0970c2675512890fabffd3d70592071d672d55cb37ade2aa8efbb8c" exitCode=0
May 25 11:25:32.328378 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.327960    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56" event={"ID":"c6148c57-736c-48a4-8dfc-d67827349aa1","Type":"ContainerDied","Data":"2723d096b0970c2675512890fabffd3d70592071d672d55cb37ade2aa8efbb8c"}
May 25 11:25:32.329908 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.329885    2566 generic.go:358] "Generic (PLEG): container finished" podID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerID="f0e691720738f22377a9353766519e7c34cb0f056459d90b399a2e7b6233d959" exitCode=0
May 25 11:25:32.330000 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.329966    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg" event={"ID":"afa3fc5a-62e9-47fe-82c9-71444b9c7f43","Type":"ContainerDied","Data":"f0e691720738f22377a9353766519e7c34cb0f056459d90b399a2e7b6233d959"}
May 25 11:25:32.331490 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.331436    2566 generic.go:358] "Generic (PLEG): container finished" podID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerID="3747a251e32e73615e48d637daf796e4173c02b2b0e12c4a2a1115a7d631bfb4" exitCode=0
May 25 11:25:32.331617 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.331522    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp" event={"ID":"76914f61-7410-46e4-a9d5-45fc74fb752b","Type":"ContainerDied","Data":"3747a251e32e73615e48d637daf796e4173c02b2b0e12c4a2a1115a7d631bfb4"}
May 25 11:25:32.333361 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.333343    2566 generic.go:358] "Generic (PLEG): container finished" podID="d5d5e079-8916-41de-befb-ccef9687add1" containerID="7bfdbc1235d646ad0db91ad080db3b68a910613bad5a20c58b42fc0143013111" exitCode=0
May 25 11:25:32.333430 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:32.333373    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq" event={"ID":"d5d5e079-8916-41de-befb-ccef9687add1","Type":"ContainerDied","Data":"7bfdbc1235d646ad0db91ad080db3b68a910613bad5a20c58b42fc0143013111"}
May 25 11:25:33.339420 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.339393    2566 generic.go:358] "Generic (PLEG): container finished" podID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerID="dec3029bea8819ac06b978efdbf0e5a9b1bad2feb46dd3de23404f6cb90c7934" exitCode=0
May 25 11:25:33.339835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.339500    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56" event={"ID":"c6148c57-736c-48a4-8dfc-d67827349aa1","Type":"ContainerDied","Data":"dec3029bea8819ac06b978efdbf0e5a9b1bad2feb46dd3de23404f6cb90c7934"}
May 25 11:25:33.341167 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.341143    2566 generic.go:358] "Generic (PLEG): container finished" podID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerID="905ed08a8322bc9632ef0f44274ea6ece2c7f3440edf3ebb568fe17fa7a37c0f" exitCode=0
May 25 11:25:33.341291 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.341203    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp" event={"ID":"76914f61-7410-46e4-a9d5-45fc74fb752b","Type":"ContainerDied","Data":"905ed08a8322bc9632ef0f44274ea6ece2c7f3440edf3ebb568fe17fa7a37c0f"}
May 25 11:25:33.342935 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.342914    2566 generic.go:358] "Generic (PLEG): container finished" podID="d5d5e079-8916-41de-befb-ccef9687add1" containerID="5b67886d671acfc450cd47e93cf6812b7e8a2d5d7dd3b61dc2628cac200ffbe3" exitCode=0
May 25 11:25:33.343018 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.342948    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq" event={"ID":"d5d5e079-8916-41de-befb-ccef9687add1","Type":"ContainerDied","Data":"5b67886d671acfc450cd47e93cf6812b7e8a2d5d7dd3b61dc2628cac200ffbe3"}
May 25 11:25:33.466823 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.466803    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:33.598227 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.598138    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-util\") pod \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") "
May 25 11:25:33.598227 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.598198    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-bundle\") pod \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") "
May 25 11:25:33.598227 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.598225    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7stvl\" (UniqueName: \"kubernetes.io/projected/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-kube-api-access-7stvl\") pod \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\" (UID: \"afa3fc5a-62e9-47fe-82c9-71444b9c7f43\") "
May 25 11:25:33.598688 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.598665    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-bundle" (OuterVolumeSpecName: "bundle") pod "afa3fc5a-62e9-47fe-82c9-71444b9c7f43" (UID: "afa3fc5a-62e9-47fe-82c9-71444b9c7f43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:33.600272 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.600247    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-kube-api-access-7stvl" (OuterVolumeSpecName: "kube-api-access-7stvl") pod "afa3fc5a-62e9-47fe-82c9-71444b9c7f43" (UID: "afa3fc5a-62e9-47fe-82c9-71444b9c7f43"). InnerVolumeSpecName "kube-api-access-7stvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:33.603615 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.603591    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-util" (OuterVolumeSpecName: "util") pod "afa3fc5a-62e9-47fe-82c9-71444b9c7f43" (UID: "afa3fc5a-62e9-47fe-82c9-71444b9c7f43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:33.699104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.699061    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:33.699104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.699095    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:33.699104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:33.699110    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7stvl\" (UniqueName: \"kubernetes.io/projected/afa3fc5a-62e9-47fe-82c9-71444b9c7f43-kube-api-access-7stvl\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.352645 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.352615    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg"
May 25 11:25:34.352645 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.352628    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767btr2pg" event={"ID":"afa3fc5a-62e9-47fe-82c9-71444b9c7f43","Type":"ContainerDied","Data":"243e1206347784d2efe7d5dcaf7af7910564da4a988ebc92ff952515315cf4cc"}
May 25 11:25:34.353171 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.352664    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="243e1206347784d2efe7d5dcaf7af7910564da4a988ebc92ff952515315cf4cc"
May 25 11:25:34.502197 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.502163    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:34.541969 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.541947    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:34.546009 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.545987    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:34.607063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.606982    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hptph\" (UniqueName: \"kubernetes.io/projected/d5d5e079-8916-41de-befb-ccef9687add1-kube-api-access-hptph\") pod \"d5d5e079-8916-41de-befb-ccef9687add1\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") "
May 25 11:25:34.607063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607024    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww45h\" (UniqueName: \"kubernetes.io/projected/76914f61-7410-46e4-a9d5-45fc74fb752b-kube-api-access-ww45h\") pod \"76914f61-7410-46e4-a9d5-45fc74fb752b\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") "
May 25 11:25:34.607063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607058    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-bundle\") pod \"76914f61-7410-46e4-a9d5-45fc74fb752b\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") "
May 25 11:25:34.607331 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607089    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-bundle\") pod \"d5d5e079-8916-41de-befb-ccef9687add1\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") "
May 25 11:25:34.607331 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607116    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-util\") pod \"d5d5e079-8916-41de-befb-ccef9687add1\" (UID: \"d5d5e079-8916-41de-befb-ccef9687add1\") "
May 25 11:25:34.607331 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607154    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-bundle\") pod \"c6148c57-736c-48a4-8dfc-d67827349aa1\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") "
May 25 11:25:34.607331 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607192    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-util\") pod \"c6148c57-736c-48a4-8dfc-d67827349aa1\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") "
May 25 11:25:34.607331 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607253    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8pxf\" (UniqueName: \"kubernetes.io/projected/c6148c57-736c-48a4-8dfc-d67827349aa1-kube-api-access-d8pxf\") pod \"c6148c57-736c-48a4-8dfc-d67827349aa1\" (UID: \"c6148c57-736c-48a4-8dfc-d67827349aa1\") "
May 25 11:25:34.607331 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607279    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-util\") pod \"76914f61-7410-46e4-a9d5-45fc74fb752b\" (UID: \"76914f61-7410-46e4-a9d5-45fc74fb752b\") "
May 25 11:25:34.607905 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607875    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-bundle" (OuterVolumeSpecName: "bundle") pod "76914f61-7410-46e4-a9d5-45fc74fb752b" (UID: "76914f61-7410-46e4-a9d5-45fc74fb752b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:34.608032 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.607918    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-bundle" (OuterVolumeSpecName: "bundle") pod "d5d5e079-8916-41de-befb-ccef9687add1" (UID: "d5d5e079-8916-41de-befb-ccef9687add1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:34.608105 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.608078    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-bundle" (OuterVolumeSpecName: "bundle") pod "c6148c57-736c-48a4-8dfc-d67827349aa1" (UID: "c6148c57-736c-48a4-8dfc-d67827349aa1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:34.609387 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.609367    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76914f61-7410-46e4-a9d5-45fc74fb752b-kube-api-access-ww45h" (OuterVolumeSpecName: "kube-api-access-ww45h") pod "76914f61-7410-46e4-a9d5-45fc74fb752b" (UID: "76914f61-7410-46e4-a9d5-45fc74fb752b"). InnerVolumeSpecName "kube-api-access-ww45h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:34.609652 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.609623    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d5e079-8916-41de-befb-ccef9687add1-kube-api-access-hptph" (OuterVolumeSpecName: "kube-api-access-hptph") pod "d5d5e079-8916-41de-befb-ccef9687add1" (UID: "d5d5e079-8916-41de-befb-ccef9687add1"). InnerVolumeSpecName "kube-api-access-hptph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:34.609813 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.609791    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6148c57-736c-48a4-8dfc-d67827349aa1-kube-api-access-d8pxf" (OuterVolumeSpecName: "kube-api-access-d8pxf") pod "c6148c57-736c-48a4-8dfc-d67827349aa1" (UID: "c6148c57-736c-48a4-8dfc-d67827349aa1"). InnerVolumeSpecName "kube-api-access-d8pxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:34.613042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.613017    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-util" (OuterVolumeSpecName: "util") pod "76914f61-7410-46e4-a9d5-45fc74fb752b" (UID: "76914f61-7410-46e4-a9d5-45fc74fb752b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:34.708113 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.708075    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.708113 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.708107    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8pxf\" (UniqueName: \"kubernetes.io/projected/c6148c57-736c-48a4-8dfc-d67827349aa1-kube-api-access-d8pxf\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.708113 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.708122    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.708348 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.708135    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hptph\" (UniqueName: \"kubernetes.io/projected/d5d5e079-8916-41de-befb-ccef9687add1-kube-api-access-hptph\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.708348 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.708146    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww45h\" (UniqueName: \"kubernetes.io/projected/76914f61-7410-46e4-a9d5-45fc74fb752b-kube-api-access-ww45h\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.708348 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.708159    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76914f61-7410-46e4-a9d5-45fc74fb752b-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.708348 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.708170    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-bundle\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.751418 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.751374    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-util" (OuterVolumeSpecName: "util") pod "c6148c57-736c-48a4-8dfc-d67827349aa1" (UID: "c6148c57-736c-48a4-8dfc-d67827349aa1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:34.751799 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.751779    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-util" (OuterVolumeSpecName: "util") pod "d5d5e079-8916-41de-befb-ccef9687add1" (UID: "d5d5e079-8916-41de-befb-ccef9687add1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:34.808980 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.808948    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5d5e079-8916-41de-befb-ccef9687add1-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:34.808980 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:34.808979    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6148c57-736c-48a4-8dfc-d67827349aa1-util\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:25:35.357943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.357915    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56"
May 25 11:25:35.358335 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.357910    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78j8j56" event={"ID":"c6148c57-736c-48a4-8dfc-d67827349aa1","Type":"ContainerDied","Data":"ec96a8b6fd2c540e18861a617a5a4e7f8c3579da5d29f89881b9e7a0f9a1c7a8"}
May 25 11:25:35.358335 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.358027    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec96a8b6fd2c540e18861a617a5a4e7f8c3579da5d29f89881b9e7a0f9a1c7a8"
May 25 11:25:35.359662 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.359640    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp" event={"ID":"76914f61-7410-46e4-a9d5-45fc74fb752b","Type":"ContainerDied","Data":"0e614b6dd189ca0b71b13bf28727d6b688fb60bc6ada21977ec5d532fce1dda7"}
May 25 11:25:35.359662 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.359663    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e614b6dd189ca0b71b13bf28727d6b688fb60bc6ada21977ec5d532fce1dda7"
May 25 11:25:35.359813 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.359687    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88mrzqp"
May 25 11:25:35.361287 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.361263    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq" event={"ID":"d5d5e079-8916-41de-befb-ccef9687add1","Type":"ContainerDied","Data":"a27bf0dd44fae4c5b0a3cc09cd97f29f8d617019f3af71165baf7516469f3185"}
May 25 11:25:35.361374 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.361291    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a27bf0dd44fae4c5b0a3cc09cd97f29f8d617019f3af71165baf7516469f3185"
May 25 11:25:35.361374 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:35.361275    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30xt8jq"
May 25 11:25:41.029360 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029325    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"]
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029628    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029639    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029649    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029655    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029662    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029668    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029675    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029681    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029688    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d5e079-8916-41de-befb-ccef9687add1" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029693    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d5e079-8916-41de-befb-ccef9687add1" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029698    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029704    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerName="util"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029711    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029716    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029723    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029728    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029733    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d5e079-8916-41de-befb-ccef9687add1" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029737    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d5e079-8916-41de-befb-ccef9687add1" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029744    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029749    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029755    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d5e079-8916-41de-befb-ccef9687add1" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029760    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d5e079-8916-41de-befb-ccef9687add1" containerName="pull"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029768    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029773    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029810    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="afa3fc5a-62e9-47fe-82c9-71444b9c7f43" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029818    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5d5e079-8916-41de-befb-ccef9687add1" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029823    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="76914f61-7410-46e4-a9d5-45fc74fb752b" containerName="extract"
May 25 11:25:41.029828 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.029831    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="c6148c57-736c-48a4-8dfc-d67827349aa1" containerName="extract"
May 25 11:25:41.032586 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.032570    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"
May 25 11:25:41.035279 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.035259    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vdmss\""
May 25 11:25:41.035945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.035928    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 25 11:25:41.035945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.035937    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 25 11:25:41.043113 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.043091    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"]
May 25 11:25:41.165306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.165262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcgns\" (UniqueName: \"kubernetes.io/projected/73d1ac66-893f-4b04-9ab9-91f39e4f902f-kube-api-access-gcgns\") pod \"limitador-operator-controller-manager-c7fb4c8d5-4fgrj\" (UID: \"73d1ac66-893f-4b04-9ab9-91f39e4f902f\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"
May 25 11:25:41.266334 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.266291    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gcgns\" (UniqueName: \"kubernetes.io/projected/73d1ac66-893f-4b04-9ab9-91f39e4f902f-kube-api-access-gcgns\") pod \"limitador-operator-controller-manager-c7fb4c8d5-4fgrj\" (UID: \"73d1ac66-893f-4b04-9ab9-91f39e4f902f\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"
May 25 11:25:41.278709 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.278677    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcgns\" (UniqueName: \"kubernetes.io/projected/73d1ac66-893f-4b04-9ab9-91f39e4f902f-kube-api-access-gcgns\") pod \"limitador-operator-controller-manager-c7fb4c8d5-4fgrj\" (UID: \"73d1ac66-893f-4b04-9ab9-91f39e4f902f\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"
May 25 11:25:41.342715 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.342683    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"
May 25 11:25:41.486170 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:41.486133    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"]
May 25 11:25:41.489279 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:41.489246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d1ac66_893f_4b04_9ab9_91f39e4f902f.slice/crio-a17d7945f5f7b1a53c65489e933ed229ee2483cea84674009c9e641ec014f17f WatchSource:0}: Error finding container a17d7945f5f7b1a53c65489e933ed229ee2483cea84674009c9e641ec014f17f: Status 404 returned error can't find the container with id a17d7945f5f7b1a53c65489e933ed229ee2483cea84674009c9e641ec014f17f
May 25 11:25:42.384689 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:42.384654    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj" event={"ID":"73d1ac66-893f-4b04-9ab9-91f39e4f902f","Type":"ContainerStarted","Data":"a17d7945f5f7b1a53c65489e933ed229ee2483cea84674009c9e641ec014f17f"}
May 25 11:25:44.360529 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.360492    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"]
May 25 11:25:44.364100 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.364079    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.366623 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.366592    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zlskw\""
May 25 11:25:44.379776 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.379751    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"]
May 25 11:25:44.494513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.494472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41b76363-18eb-4414-aab5-1a922915fcc9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-wqt9m\" (UID: \"41b76363-18eb-4414-aab5-1a922915fcc9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.494690 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.494578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7dcr\" (UniqueName: \"kubernetes.io/projected/41b76363-18eb-4414-aab5-1a922915fcc9-kube-api-access-c7dcr\") pod \"kuadrant-operator-controller-manager-57dd85bb4-wqt9m\" (UID: \"41b76363-18eb-4414-aab5-1a922915fcc9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.595532 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.595498    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7dcr\" (UniqueName: \"kubernetes.io/projected/41b76363-18eb-4414-aab5-1a922915fcc9-kube-api-access-c7dcr\") pod \"kuadrant-operator-controller-manager-57dd85bb4-wqt9m\" (UID: \"41b76363-18eb-4414-aab5-1a922915fcc9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.595678 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.595590    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41b76363-18eb-4414-aab5-1a922915fcc9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-wqt9m\" (UID: \"41b76363-18eb-4414-aab5-1a922915fcc9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.595954 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.595935    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41b76363-18eb-4414-aab5-1a922915fcc9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-wqt9m\" (UID: \"41b76363-18eb-4414-aab5-1a922915fcc9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.607705 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.607676    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7dcr\" (UniqueName: \"kubernetes.io/projected/41b76363-18eb-4414-aab5-1a922915fcc9-kube-api-access-c7dcr\") pod \"kuadrant-operator-controller-manager-57dd85bb4-wqt9m\" (UID: \"41b76363-18eb-4414-aab5-1a922915fcc9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.676390 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.676302    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:44.828850 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:44.828757    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"]
May 25 11:25:44.831655 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:25:44.831625    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b76363_18eb_4414_aab5_1a922915fcc9.slice/crio-15ba9f2b0f9810944da4b2802d3a22bb1688586149c1f01d2e7e9cc58dd1bf3b WatchSource:0}: Error finding container 15ba9f2b0f9810944da4b2802d3a22bb1688586149c1f01d2e7e9cc58dd1bf3b: Status 404 returned error can't find the container with id 15ba9f2b0f9810944da4b2802d3a22bb1688586149c1f01d2e7e9cc58dd1bf3b
May 25 11:25:45.404477 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:45.404440    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj" event={"ID":"73d1ac66-893f-4b04-9ab9-91f39e4f902f","Type":"ContainerStarted","Data":"9d1ce2326a55834521afed1b0de670ff4d7dfd9088373c417d15192fc0c89198"}
May 25 11:25:45.404936 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:45.404526    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"
May 25 11:25:45.405634 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:45.405608    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m" event={"ID":"41b76363-18eb-4414-aab5-1a922915fcc9","Type":"ContainerStarted","Data":"15ba9f2b0f9810944da4b2802d3a22bb1688586149c1f01d2e7e9cc58dd1bf3b"}
May 25 11:25:45.423738 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:45.423692    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj" podStartSLOduration=2.389517525 podStartE2EDuration="5.423675065s" podCreationTimestamp="2026-05-25 11:25:40 +0000 UTC" firstStartedPulling="2026-05-25 11:25:41.49159612 +0000 UTC m=+500.246572895" lastFinishedPulling="2026-05-25 11:25:44.525753657 +0000 UTC m=+503.280730435" observedRunningTime="2026-05-25 11:25:45.421850939 +0000 UTC m=+504.176827749" watchObservedRunningTime="2026-05-25 11:25:45.423675065 +0000 UTC m=+504.178651865"
May 25 11:25:53.436158 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:53.436117    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m" event={"ID":"41b76363-18eb-4414-aab5-1a922915fcc9","Type":"ContainerStarted","Data":"c7d6e536c4c4a47b65c721baaf9eb818f605dd0a98eccf1903df5f3d8224c265"}
May 25 11:25:53.436635 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:53.436336    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:25:53.460398 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:53.460348    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m" podStartSLOduration=1.784493804 podStartE2EDuration="9.460334565s" podCreationTimestamp="2026-05-25 11:25:44 +0000 UTC" firstStartedPulling="2026-05-25 11:25:44.834076601 +0000 UTC m=+503.589053379" lastFinishedPulling="2026-05-25 11:25:52.509917363 +0000 UTC m=+511.264894140" observedRunningTime="2026-05-25 11:25:53.458759173 +0000 UTC m=+512.213735972" watchObservedRunningTime="2026-05-25 11:25:53.460334565 +0000 UTC m=+512.215311360"
May 25 11:25:56.412365 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:25:56.412332    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-4fgrj"
May 25 11:26:04.444008 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:04.443978    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-wqt9m"
May 25 11:26:38.804206 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.804126    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-lsvdj"]
May 25 11:26:38.806367 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.806348    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:38.808656 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.808637    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 25 11:26:38.808771 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.808639    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rv2jq\""
May 25 11:26:38.815686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.815663    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-lsvdj"]
May 25 11:26:38.845561 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.845514    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-lsvdj"]
May 25 11:26:38.924210 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.924170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82c229ca-664f-4254-80de-5946e4940ae6-config-file\") pod \"limitador-limitador-67566c68b4-lsvdj\" (UID: \"82c229ca-664f-4254-80de-5946e4940ae6\") " pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:38.924210 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:38.924213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hbt\" (UniqueName: \"kubernetes.io/projected/82c229ca-664f-4254-80de-5946e4940ae6-kube-api-access-q6hbt\") pod \"limitador-limitador-67566c68b4-lsvdj\" (UID: \"82c229ca-664f-4254-80de-5946e4940ae6\") " pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:39.025600 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:39.025558    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82c229ca-664f-4254-80de-5946e4940ae6-config-file\") pod \"limitador-limitador-67566c68b4-lsvdj\" (UID: \"82c229ca-664f-4254-80de-5946e4940ae6\") " pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:39.025765 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:39.025606    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hbt\" (UniqueName: \"kubernetes.io/projected/82c229ca-664f-4254-80de-5946e4940ae6-kube-api-access-q6hbt\") pod \"limitador-limitador-67566c68b4-lsvdj\" (UID: \"82c229ca-664f-4254-80de-5946e4940ae6\") " pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:39.026143 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:39.026124    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82c229ca-664f-4254-80de-5946e4940ae6-config-file\") pod \"limitador-limitador-67566c68b4-lsvdj\" (UID: \"82c229ca-664f-4254-80de-5946e4940ae6\") " pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:39.034113 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:39.034087    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hbt\" (UniqueName: \"kubernetes.io/projected/82c229ca-664f-4254-80de-5946e4940ae6-kube-api-access-q6hbt\") pod \"limitador-limitador-67566c68b4-lsvdj\" (UID: \"82c229ca-664f-4254-80de-5946e4940ae6\") " pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:39.117256 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:39.117221    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:39.247751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:39.247719    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-lsvdj"]
May 25 11:26:39.251028 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:26:39.250996    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c229ca_664f_4254_80de_5946e4940ae6.slice/crio-2301fc2f881eb667bd206bcc787a5affe305f358cea31172eae1ac8c77211f83 WatchSource:0}: Error finding container 2301fc2f881eb667bd206bcc787a5affe305f358cea31172eae1ac8c77211f83: Status 404 returned error can't find the container with id 2301fc2f881eb667bd206bcc787a5affe305f358cea31172eae1ac8c77211f83
May 25 11:26:39.596578 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:39.596523    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj" event={"ID":"82c229ca-664f-4254-80de-5946e4940ae6","Type":"ContainerStarted","Data":"2301fc2f881eb667bd206bcc787a5affe305f358cea31172eae1ac8c77211f83"}
May 25 11:26:43.616958 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:43.616874    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj" event={"ID":"82c229ca-664f-4254-80de-5946e4940ae6","Type":"ContainerStarted","Data":"e6b4ca3fba6d6daf65fd54111d750543652effdb033f850b2c0687a2f48e69ab"}
May 25 11:26:43.617336 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:43.617097    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:26:43.634885 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:43.634825    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj" podStartSLOduration=1.533495597 podStartE2EDuration="5.63480812s" podCreationTimestamp="2026-05-25 11:26:38 +0000 UTC" firstStartedPulling="2026-05-25 11:26:39.252780538 +0000 UTC m=+558.007757313" lastFinishedPulling="2026-05-25 11:26:43.354093058 +0000 UTC m=+562.109069836" observedRunningTime="2026-05-25 11:26:43.632360578 +0000 UTC m=+562.387337375" watchObservedRunningTime="2026-05-25 11:26:43.63480812 +0000 UTC m=+562.389784916"
May 25 11:26:54.622067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:26:54.622039    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-lsvdj"
May 25 11:27:18.989172 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:18.989131    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"]
May 25 11:27:18.989739 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:18.989364    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl" podUID="405f5541-a139-48c7-95b3-8aba4b627ca5" containerName="discovery" containerID="cri-o://245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa" gracePeriod=30
May 25 11:27:19.242816 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.242745    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:27:19.244970 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.244947    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-ca-configmap\") pod \"405f5541-a139-48c7-95b3-8aba4b627ca5\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") "
May 25 11:27:19.245063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.244997    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-cacerts\") pod \"405f5541-a139-48c7-95b3-8aba4b627ca5\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") "
May 25 11:27:19.245063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.245019    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f5541-a139-48c7-95b3-8aba4b627ca5-local-certs\") pod \"405f5541-a139-48c7-95b3-8aba4b627ca5\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") "
May 25 11:27:19.245063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.245044    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-dns-cert\") pod \"405f5541-a139-48c7-95b3-8aba4b627ca5\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") "
May 25 11:27:19.245214 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.245088    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-kubeconfig\") pod \"405f5541-a139-48c7-95b3-8aba4b627ca5\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") "
May 25 11:27:19.245214 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.245112    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7bmx\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-kube-api-access-g7bmx\") pod \"405f5541-a139-48c7-95b3-8aba4b627ca5\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") "
May 25 11:27:19.245306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.245230    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-token\") pod \"405f5541-a139-48c7-95b3-8aba4b627ca5\" (UID: \"405f5541-a139-48c7-95b3-8aba4b627ca5\") "
May 25 11:27:19.245596 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.245529    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "405f5541-a139-48c7-95b3-8aba4b627ca5" (UID: "405f5541-a139-48c7-95b3-8aba4b627ca5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:27:19.247921 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.247888    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "405f5541-a139-48c7-95b3-8aba4b627ca5" (UID: "405f5541-a139-48c7-95b3-8aba4b627ca5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:27:19.248054 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.248014    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "405f5541-a139-48c7-95b3-8aba4b627ca5" (UID: "405f5541-a139-48c7-95b3-8aba4b627ca5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:27:19.248151 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.248123    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-cacerts" (OuterVolumeSpecName: "cacerts") pod "405f5541-a139-48c7-95b3-8aba4b627ca5" (UID: "405f5541-a139-48c7-95b3-8aba4b627ca5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:27:19.248259 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.248182    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f5541-a139-48c7-95b3-8aba4b627ca5-local-certs" (OuterVolumeSpecName: "local-certs") pod "405f5541-a139-48c7-95b3-8aba4b627ca5" (UID: "405f5541-a139-48c7-95b3-8aba4b627ca5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:27:19.248305 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.248277    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-kube-api-access-g7bmx" (OuterVolumeSpecName: "kube-api-access-g7bmx") pod "405f5541-a139-48c7-95b3-8aba4b627ca5" (UID: "405f5541-a139-48c7-95b3-8aba4b627ca5"). InnerVolumeSpecName "kube-api-access-g7bmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:27:19.248305 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.248276    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-token" (OuterVolumeSpecName: "istio-token") pod "405f5541-a139-48c7-95b3-8aba4b627ca5" (UID: "405f5541-a139-48c7-95b3-8aba4b627ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:27:19.345769 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.345734    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f5541-a139-48c7-95b3-8aba4b627ca5-local-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:27:19.345769 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.345766    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-dns-cert\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:27:19.345979 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.345781    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-kubeconfig\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:27:19.345979 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.345792    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7bmx\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-kube-api-access-g7bmx\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:27:19.345979 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.345801    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-token\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:27:19.345979 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.345809    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/405f5541-a139-48c7-95b3-8aba4b627ca5-istio-csr-ca-configmap\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:27:19.345979 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.345818    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/405f5541-a139-48c7-95b3-8aba4b627ca5-cacerts\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:27:19.743705 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.743664    2566 generic.go:358] "Generic (PLEG): container finished" podID="405f5541-a139-48c7-95b3-8aba4b627ca5" containerID="245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa" exitCode=0
May 25 11:27:19.743888 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.743734    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"
May 25 11:27:19.743888 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.743749    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl" event={"ID":"405f5541-a139-48c7-95b3-8aba4b627ca5","Type":"ContainerDied","Data":"245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa"}
May 25 11:27:19.743888 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.743789    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl" event={"ID":"405f5541-a139-48c7-95b3-8aba4b627ca5","Type":"ContainerDied","Data":"21c32ab6006eae425d9a825b42b5beb6f0a487fd835cfd774a7141af1d775786"}
May 25 11:27:19.743888 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.743805    2566 scope.go:117] "RemoveContainer" containerID="245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa"
May 25 11:27:19.753116 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.753095    2566 scope.go:117] "RemoveContainer" containerID="245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa"
May 25 11:27:19.753419 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:27:19.753403    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa\": container with ID starting with 245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa not found: ID does not exist" containerID="245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa"
May 25 11:27:19.753482 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.753426    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa"} err="failed to get container status \"245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa\": rpc error: code = NotFound desc = could not find container \"245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa\": container with ID starting with 245b1f8b560b704bb370fb47661cfbd7983ed623538fe597038aa7449ced40fa not found: ID does not exist"
May 25 11:27:19.766084 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.766046    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"]
May 25 11:27:19.772074 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.772043    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-g9wbl"]
May 25 11:27:19.835579 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:19.835527    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="405f5541-a139-48c7-95b3-8aba4b627ca5" path="/var/lib/kubelet/pods/405f5541-a139-48c7-95b3-8aba4b627ca5/volumes"
May 25 11:27:24.671714 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.671678    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-gbzfb"]
May 25 11:27:24.672162 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.672001    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405f5541-a139-48c7-95b3-8aba4b627ca5" containerName="discovery"
May 25 11:27:24.672162 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.672013    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="405f5541-a139-48c7-95b3-8aba4b627ca5" containerName="discovery"
May 25 11:27:24.672162 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.672078    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="405f5541-a139-48c7-95b3-8aba4b627ca5" containerName="discovery"
May 25 11:27:24.676535 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.676515    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:24.679470 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.679445    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-drmzr\""
May 25 11:27:24.680035 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.680021    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 11:27:24.680084 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.680061    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 11:27:24.680247 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.680234    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 11:27:24.680893 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.680873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5cb138-c519-425c-b1dc-7311ae3b8df4-cert\") pod \"kserve-controller-manager-559cc6dd6f-gbzfb\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:24.680946 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.680914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjpg\" (UniqueName: \"kubernetes.io/projected/5f5cb138-c519-425c-b1dc-7311ae3b8df4-kube-api-access-7wjpg\") pod \"kserve-controller-manager-559cc6dd6f-gbzfb\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:24.685196 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.685172    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-gbzfb"]
May 25 11:27:24.688616 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.688598    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"]
May 25 11:27:24.691725 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.691700    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:24.693999 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.693979    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 25 11:27:24.694183 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.694169    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rbnw8\""
May 25 11:27:24.702635 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.702614    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"]
May 25 11:27:24.716756 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.716731    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-2t78h"]
May 25 11:27:24.719913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.719892    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:24.722273 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.722254    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2b4gg\""
May 25 11:27:24.722377 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.722292    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 11:27:24.730219 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.730192    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-2t78h"]
May 25 11:27:24.781349 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.781312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkskv\" (UniqueName: \"kubernetes.io/projected/8196380f-10fd-41c7-86e4-87af8dfa4179-kube-api-access-lkskv\") pod \"seaweedfs-79f457656b-2t78h\" (UID: \"8196380f-10fd-41c7-86e4-87af8dfa4179\") " pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:24.781349 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.781363    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5cb138-c519-425c-b1dc-7311ae3b8df4-cert\") pod \"kserve-controller-manager-559cc6dd6f-gbzfb\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:24.781589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.781411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f144da74-d67f-4b88-9ea1-37d100500c91-cert\") pod \"llmisvc-controller-manager-c557f5ddb-xwbrg\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") " pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:24.781589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.781559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8196380f-10fd-41c7-86e4-87af8dfa4179-data\") pod \"seaweedfs-79f457656b-2t78h\" (UID: \"8196380f-10fd-41c7-86e4-87af8dfa4179\") " pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:24.781665 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.781597    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjpg\" (UniqueName: \"kubernetes.io/projected/5f5cb138-c519-425c-b1dc-7311ae3b8df4-kube-api-access-7wjpg\") pod \"kserve-controller-manager-559cc6dd6f-gbzfb\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:24.781665 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.781651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgv8\" (UniqueName: \"kubernetes.io/projected/f144da74-d67f-4b88-9ea1-37d100500c91-kube-api-access-6xgv8\") pod \"llmisvc-controller-manager-c557f5ddb-xwbrg\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") " pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:24.783770 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.783751    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5cb138-c519-425c-b1dc-7311ae3b8df4-cert\") pod \"kserve-controller-manager-559cc6dd6f-gbzfb\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:24.792314 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.792288    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjpg\" (UniqueName: \"kubernetes.io/projected/5f5cb138-c519-425c-b1dc-7311ae3b8df4-kube-api-access-7wjpg\") pod \"kserve-controller-manager-559cc6dd6f-gbzfb\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:24.882482 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.882452    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8196380f-10fd-41c7-86e4-87af8dfa4179-data\") pod \"seaweedfs-79f457656b-2t78h\" (UID: \"8196380f-10fd-41c7-86e4-87af8dfa4179\") " pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:24.882712 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.882527    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgv8\" (UniqueName: \"kubernetes.io/projected/f144da74-d67f-4b88-9ea1-37d100500c91-kube-api-access-6xgv8\") pod \"llmisvc-controller-manager-c557f5ddb-xwbrg\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") " pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:24.882712 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.882588    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkskv\" (UniqueName: \"kubernetes.io/projected/8196380f-10fd-41c7-86e4-87af8dfa4179-kube-api-access-lkskv\") pod \"seaweedfs-79f457656b-2t78h\" (UID: \"8196380f-10fd-41c7-86e4-87af8dfa4179\") " pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:24.882712 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.882635    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f144da74-d67f-4b88-9ea1-37d100500c91-cert\") pod \"llmisvc-controller-manager-c557f5ddb-xwbrg\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") " pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:24.882882 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.882865    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8196380f-10fd-41c7-86e4-87af8dfa4179-data\") pod \"seaweedfs-79f457656b-2t78h\" (UID: \"8196380f-10fd-41c7-86e4-87af8dfa4179\") " pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:24.885005 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.884983    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f144da74-d67f-4b88-9ea1-37d100500c91-cert\") pod \"llmisvc-controller-manager-c557f5ddb-xwbrg\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") " pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:24.892367 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.892346    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgv8\" (UniqueName: \"kubernetes.io/projected/f144da74-d67f-4b88-9ea1-37d100500c91-kube-api-access-6xgv8\") pod \"llmisvc-controller-manager-c557f5ddb-xwbrg\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") " pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:24.892470 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.892446    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkskv\" (UniqueName: \"kubernetes.io/projected/8196380f-10fd-41c7-86e4-87af8dfa4179-kube-api-access-lkskv\") pod \"seaweedfs-79f457656b-2t78h\" (UID: \"8196380f-10fd-41c7-86e4-87af8dfa4179\") " pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:24.988431 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:24.988345    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:25.002103 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.002075    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:25.029779 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.029730    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:25.135860 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.135679    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-gbzfb"]
May 25 11:27:25.138589 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:27:25.138555    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5cb138_c519_425c_b1dc_7311ae3b8df4.slice/crio-798b6f8ac89889e82ddc51265dd0a86de189f8410afd4c360656374cb72ae6f7 WatchSource:0}: Error finding container 798b6f8ac89889e82ddc51265dd0a86de189f8410afd4c360656374cb72ae6f7: Status 404 returned error can't find the container with id 798b6f8ac89889e82ddc51265dd0a86de189f8410afd4c360656374cb72ae6f7
May 25 11:27:25.160489 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.160464    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"]
May 25 11:27:25.163165 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:27:25.163132    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf144da74_d67f_4b88_9ea1_37d100500c91.slice/crio-a5fc621f63b200433ec2b523ec048ebb8da39822126b189543d53e3fae179f3a WatchSource:0}: Error finding container a5fc621f63b200433ec2b523ec048ebb8da39822126b189543d53e3fae179f3a: Status 404 returned error can't find the container with id a5fc621f63b200433ec2b523ec048ebb8da39822126b189543d53e3fae179f3a
May 25 11:27:25.186822 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.186793    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-2t78h"]
May 25 11:27:25.189363 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:27:25.189333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8196380f_10fd_41c7_86e4_87af8dfa4179.slice/crio-1f5df78024d4df9b5614a545c2bb2e75192e52e57a7989227ad12da4d29894f6 WatchSource:0}: Error finding container 1f5df78024d4df9b5614a545c2bb2e75192e52e57a7989227ad12da4d29894f6: Status 404 returned error can't find the container with id 1f5df78024d4df9b5614a545c2bb2e75192e52e57a7989227ad12da4d29894f6
May 25 11:27:25.766787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.766748    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg" event={"ID":"f144da74-d67f-4b88-9ea1-37d100500c91","Type":"ContainerStarted","Data":"a5fc621f63b200433ec2b523ec048ebb8da39822126b189543d53e3fae179f3a"}
May 25 11:27:25.768660 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.768621    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb" event={"ID":"5f5cb138-c519-425c-b1dc-7311ae3b8df4","Type":"ContainerStarted","Data":"798b6f8ac89889e82ddc51265dd0a86de189f8410afd4c360656374cb72ae6f7"}
May 25 11:27:25.769928 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:25.769897    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-2t78h" event={"ID":"8196380f-10fd-41c7-86e4-87af8dfa4179","Type":"ContainerStarted","Data":"1f5df78024d4df9b5614a545c2bb2e75192e52e57a7989227ad12da4d29894f6"}
May 25 11:27:30.791269 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.791227    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb" event={"ID":"5f5cb138-c519-425c-b1dc-7311ae3b8df4","Type":"ContainerStarted","Data":"221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e"}
May 25 11:27:30.791761 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.791288    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:27:30.792700 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.792672    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-2t78h" event={"ID":"8196380f-10fd-41c7-86e4-87af8dfa4179","Type":"ContainerStarted","Data":"320d5a1976ffb901396e564bd63ccff3c164f1ec55ad2261654f3eaaa4b55712"}
May 25 11:27:30.792788 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.792713    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:27:30.793933 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.793912    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg" event={"ID":"f144da74-d67f-4b88-9ea1-37d100500c91","Type":"ContainerStarted","Data":"87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be"}
May 25 11:27:30.794057 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.794044    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:27:30.830835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.830784    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb" podStartSLOduration=2.89361188 podStartE2EDuration="6.830767125s" podCreationTimestamp="2026-05-25 11:27:24 +0000 UTC" firstStartedPulling="2026-05-25 11:27:25.140201012 +0000 UTC m=+603.895177801" lastFinishedPulling="2026-05-25 11:27:29.077356269 +0000 UTC m=+607.832333046" observedRunningTime="2026-05-25 11:27:30.811460156 +0000 UTC m=+609.566436951" watchObservedRunningTime="2026-05-25 11:27:30.830767125 +0000 UTC m=+609.585743923"
May 25 11:27:30.832581 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.832528    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-2t78h" podStartSLOduration=2.254377279 podStartE2EDuration="6.832519108s" podCreationTimestamp="2026-05-25 11:27:24 +0000 UTC" firstStartedPulling="2026-05-25 11:27:25.190725323 +0000 UTC m=+603.945702097" lastFinishedPulling="2026-05-25 11:27:29.768867145 +0000 UTC m=+608.523843926" observedRunningTime="2026-05-25 11:27:30.83003669 +0000 UTC m=+609.585013485" watchObservedRunningTime="2026-05-25 11:27:30.832519108 +0000 UTC m=+609.587495903"
May 25 11:27:30.847432 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:30.847375    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg" podStartSLOduration=2.282659751 podStartE2EDuration="6.847359708s" podCreationTimestamp="2026-05-25 11:27:24 +0000 UTC" firstStartedPulling="2026-05-25 11:27:25.164491002 +0000 UTC m=+603.919467780" lastFinishedPulling="2026-05-25 11:27:29.729190963 +0000 UTC m=+608.484167737" observedRunningTime="2026-05-25 11:27:30.846823264 +0000 UTC m=+609.601800060" watchObservedRunningTime="2026-05-25 11:27:30.847359708 +0000 UTC m=+609.602336505"
May 25 11:27:36.802707 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:27:36.802674    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-2t78h"
May 25 11:28:01.802531 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:01.802497    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:28:01.805643 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:01.805621    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:28:03.088672 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.088637    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-gbzfb"]
May 25 11:28:03.089076 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.088873    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb" podUID="5f5cb138-c519-425c-b1dc-7311ae3b8df4" containerName="manager" containerID="cri-o://221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e" gracePeriod=10
May 25 11:28:03.113984 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:28:03.113946    2566 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5cb138_c519_425c_b1dc_7311ae3b8df4.slice/crio-221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e.scope\": RecentStats: unable to find data in memory cache]"
May 25 11:28:03.114748 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.114722    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-qpvw7"]
May 25 11:28:03.117389 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.117370    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.124707 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.124678    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-qpvw7"]
May 25 11:28:03.191218 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.191177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxm7z\" (UniqueName: \"kubernetes.io/projected/30f14716-eb7a-44f0-9a9f-c44a2636fbeb-kube-api-access-kxm7z\") pod \"kserve-controller-manager-559cc6dd6f-qpvw7\" (UID: \"30f14716-eb7a-44f0-9a9f-c44a2636fbeb\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.191381 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.191243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f14716-eb7a-44f0-9a9f-c44a2636fbeb-cert\") pod \"kserve-controller-manager-559cc6dd6f-qpvw7\" (UID: \"30f14716-eb7a-44f0-9a9f-c44a2636fbeb\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.292104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.292063    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kxm7z\" (UniqueName: \"kubernetes.io/projected/30f14716-eb7a-44f0-9a9f-c44a2636fbeb-kube-api-access-kxm7z\") pod \"kserve-controller-manager-559cc6dd6f-qpvw7\" (UID: \"30f14716-eb7a-44f0-9a9f-c44a2636fbeb\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.292286 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.292143    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f14716-eb7a-44f0-9a9f-c44a2636fbeb-cert\") pod \"kserve-controller-manager-559cc6dd6f-qpvw7\" (UID: \"30f14716-eb7a-44f0-9a9f-c44a2636fbeb\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.294676 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.294642    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f14716-eb7a-44f0-9a9f-c44a2636fbeb-cert\") pod \"kserve-controller-manager-559cc6dd6f-qpvw7\" (UID: \"30f14716-eb7a-44f0-9a9f-c44a2636fbeb\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.300353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.300326    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxm7z\" (UniqueName: \"kubernetes.io/projected/30f14716-eb7a-44f0-9a9f-c44a2636fbeb-kube-api-access-kxm7z\") pod \"kserve-controller-manager-559cc6dd6f-qpvw7\" (UID: \"30f14716-eb7a-44f0-9a9f-c44a2636fbeb\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.335753 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.335726    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:28:03.393534 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.393447    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5cb138-c519-425c-b1dc-7311ae3b8df4-cert\") pod \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") "
May 25 11:28:03.393715 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.393564    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wjpg\" (UniqueName: \"kubernetes.io/projected/5f5cb138-c519-425c-b1dc-7311ae3b8df4-kube-api-access-7wjpg\") pod \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\" (UID: \"5f5cb138-c519-425c-b1dc-7311ae3b8df4\") "
May 25 11:28:03.395597 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.395562    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5cb138-c519-425c-b1dc-7311ae3b8df4-cert" (OuterVolumeSpecName: "cert") pod "5f5cb138-c519-425c-b1dc-7311ae3b8df4" (UID: "5f5cb138-c519-425c-b1dc-7311ae3b8df4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:28:03.395735 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.395635    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5cb138-c519-425c-b1dc-7311ae3b8df4-kube-api-access-7wjpg" (OuterVolumeSpecName: "kube-api-access-7wjpg") pod "5f5cb138-c519-425c-b1dc-7311ae3b8df4" (UID: "5f5cb138-c519-425c-b1dc-7311ae3b8df4"). InnerVolumeSpecName "kube-api-access-7wjpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:28:03.462066 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.462027    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:03.495069 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.495031    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wjpg\" (UniqueName: \"kubernetes.io/projected/5f5cb138-c519-425c-b1dc-7311ae3b8df4-kube-api-access-7wjpg\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:28:03.495069 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.495064    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f5cb138-c519-425c-b1dc-7311ae3b8df4-cert\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:28:03.586377 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.586350    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-qpvw7"]
May 25 11:28:03.588676 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:28:03.588648    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f14716_eb7a_44f0_9a9f_c44a2636fbeb.slice/crio-e7c3404f87584121141ea68b0b4ce9db0f6326f81452e7360f648879a5129c9c WatchSource:0}: Error finding container e7c3404f87584121141ea68b0b4ce9db0f6326f81452e7360f648879a5129c9c: Status 404 returned error can't find the container with id e7c3404f87584121141ea68b0b4ce9db0f6326f81452e7360f648879a5129c9c
May 25 11:28:03.589953 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.589937    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:28:03.917475 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.917442    2566 generic.go:358] "Generic (PLEG): container finished" podID="5f5cb138-c519-425c-b1dc-7311ae3b8df4" containerID="221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e" exitCode=0
May 25 11:28:03.917650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.917511    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb"
May 25 11:28:03.917650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.917521    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb" event={"ID":"5f5cb138-c519-425c-b1dc-7311ae3b8df4","Type":"ContainerDied","Data":"221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e"}
May 25 11:28:03.917650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.917584    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-gbzfb" event={"ID":"5f5cb138-c519-425c-b1dc-7311ae3b8df4","Type":"ContainerDied","Data":"798b6f8ac89889e82ddc51265dd0a86de189f8410afd4c360656374cb72ae6f7"}
May 25 11:28:03.917650 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.917600    2566 scope.go:117] "RemoveContainer" containerID="221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e"
May 25 11:28:03.918732 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.918711    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7" event={"ID":"30f14716-eb7a-44f0-9a9f-c44a2636fbeb","Type":"ContainerStarted","Data":"e7c3404f87584121141ea68b0b4ce9db0f6326f81452e7360f648879a5129c9c"}
May 25 11:28:03.925664 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.925580    2566 scope.go:117] "RemoveContainer" containerID="221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e"
May 25 11:28:03.925868 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:28:03.925849    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e\": container with ID starting with 221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e not found: ID does not exist" containerID="221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e"
May 25 11:28:03.925977 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.925879    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e"} err="failed to get container status \"221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e\": rpc error: code = NotFound desc = could not find container \"221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e\": container with ID starting with 221e6da7d3801078ba4809a6ce4787fac7c7dcd86434f75d8b0a6a368be8ac3e not found: ID does not exist"
May 25 11:28:03.933064 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.933044    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-gbzfb"]
May 25 11:28:03.938666 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:03.938639    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-gbzfb"]
May 25 11:28:04.923769 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:04.923734    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7" event={"ID":"30f14716-eb7a-44f0-9a9f-c44a2636fbeb","Type":"ContainerStarted","Data":"bb6b57423ffcab1a2fb995cf9a51c0152395f27e2058c6ceea90ec5eefeb659e"}
May 25 11:28:04.924261 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:04.923840    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:04.942722 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:04.942665    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7" podStartSLOduration=1.656655545 podStartE2EDuration="1.942651156s" podCreationTimestamp="2026-05-25 11:28:03 +0000 UTC" firstStartedPulling="2026-05-25 11:28:03.59005577 +0000 UTC m=+642.345032543" lastFinishedPulling="2026-05-25 11:28:03.876051367 +0000 UTC m=+642.631028154" observedRunningTime="2026-05-25 11:28:04.941012575 +0000 UTC m=+643.695989365" watchObservedRunningTime="2026-05-25 11:28:04.942651156 +0000 UTC m=+643.697627951"
May 25 11:28:05.835231 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:05.835199    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5cb138-c519-425c-b1dc-7311ae3b8df4" path="/var/lib/kubelet/pods/5f5cb138-c519-425c-b1dc-7311ae3b8df4/volumes"
May 25 11:28:35.931367 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:35.931337    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-559cc6dd6f-qpvw7"
May 25 11:28:36.850659 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.850625    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-t5pkt"]
May 25 11:28:36.850927 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.850914    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5cb138-c519-425c-b1dc-7311ae3b8df4" containerName="manager"
May 25 11:28:36.850976 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.850928    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5cb138-c519-425c-b1dc-7311ae3b8df4" containerName="manager"
May 25 11:28:36.851010 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.850985    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f5cb138-c519-425c-b1dc-7311ae3b8df4" containerName="manager"
May 25 11:28:36.853994 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.853978    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:36.856395 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.856372    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 11:28:36.856508 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.856431    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-t25ft\""
May 25 11:28:36.860548 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.860511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4wr\" (UniqueName: \"kubernetes.io/projected/0a34b29f-05db-4110-ad8e-d49951a1aa73-kube-api-access-9c4wr\") pod \"model-serving-api-86f7b4b499-t5pkt\" (UID: \"0a34b29f-05db-4110-ad8e-d49951a1aa73\") " pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:36.860674 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.860648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a34b29f-05db-4110-ad8e-d49951a1aa73-tls-certs\") pod \"model-serving-api-86f7b4b499-t5pkt\" (UID: \"0a34b29f-05db-4110-ad8e-d49951a1aa73\") " pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:36.865138 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.865105    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-t5pkt"]
May 25 11:28:36.866928 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.866908    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-f6h99"]
May 25 11:28:36.870119 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.870104    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:36.872350 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.872331    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 25 11:28:36.872478 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.872422    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-kfm46\""
May 25 11:28:36.880090 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.880064    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-f6h99"]
May 25 11:28:36.961361 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.961326    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4wr\" (UniqueName: \"kubernetes.io/projected/0a34b29f-05db-4110-ad8e-d49951a1aa73-kube-api-access-9c4wr\") pod \"model-serving-api-86f7b4b499-t5pkt\" (UID: \"0a34b29f-05db-4110-ad8e-d49951a1aa73\") " pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:36.961767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.961371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdg2\" (UniqueName: \"kubernetes.io/projected/65d1e61e-0dfa-41bd-bf39-4387253f6b00-kube-api-access-msdg2\") pod \"odh-model-controller-696fc77849-f6h99\" (UID: \"65d1e61e-0dfa-41bd-bf39-4387253f6b00\") " pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:36.961767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.961405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d1e61e-0dfa-41bd-bf39-4387253f6b00-cert\") pod \"odh-model-controller-696fc77849-f6h99\" (UID: \"65d1e61e-0dfa-41bd-bf39-4387253f6b00\") " pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:36.961767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.961446    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a34b29f-05db-4110-ad8e-d49951a1aa73-tls-certs\") pod \"model-serving-api-86f7b4b499-t5pkt\" (UID: \"0a34b29f-05db-4110-ad8e-d49951a1aa73\") " pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:36.964051 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.964026    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a34b29f-05db-4110-ad8e-d49951a1aa73-tls-certs\") pod \"model-serving-api-86f7b4b499-t5pkt\" (UID: \"0a34b29f-05db-4110-ad8e-d49951a1aa73\") " pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:36.973989 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:36.973964    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4wr\" (UniqueName: \"kubernetes.io/projected/0a34b29f-05db-4110-ad8e-d49951a1aa73-kube-api-access-9c4wr\") pod \"model-serving-api-86f7b4b499-t5pkt\" (UID: \"0a34b29f-05db-4110-ad8e-d49951a1aa73\") " pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:37.062336 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.062304    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-msdg2\" (UniqueName: \"kubernetes.io/projected/65d1e61e-0dfa-41bd-bf39-4387253f6b00-kube-api-access-msdg2\") pod \"odh-model-controller-696fc77849-f6h99\" (UID: \"65d1e61e-0dfa-41bd-bf39-4387253f6b00\") " pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:37.062336 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.062344    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d1e61e-0dfa-41bd-bf39-4387253f6b00-cert\") pod \"odh-model-controller-696fc77849-f6h99\" (UID: \"65d1e61e-0dfa-41bd-bf39-4387253f6b00\") " pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:37.062579 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:28:37.062449    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 25 11:28:37.062579 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:28:37.062501    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d1e61e-0dfa-41bd-bf39-4387253f6b00-cert podName:65d1e61e-0dfa-41bd-bf39-4387253f6b00 nodeName:}" failed. No retries permitted until 2026-05-25 11:28:37.562485579 +0000 UTC m=+676.317462353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/65d1e61e-0dfa-41bd-bf39-4387253f6b00-cert") pod "odh-model-controller-696fc77849-f6h99" (UID: "65d1e61e-0dfa-41bd-bf39-4387253f6b00") : secret "odh-model-controller-webhook-cert" not found
May 25 11:28:37.071795 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.071768    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdg2\" (UniqueName: \"kubernetes.io/projected/65d1e61e-0dfa-41bd-bf39-4387253f6b00-kube-api-access-msdg2\") pod \"odh-model-controller-696fc77849-f6h99\" (UID: \"65d1e61e-0dfa-41bd-bf39-4387253f6b00\") " pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:37.165357 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.165260    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:37.292297 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.292271    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-t5pkt"]
May 25 11:28:37.293760 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:28:37.293731    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a34b29f_05db_4110_ad8e_d49951a1aa73.slice/crio-a47f1768b776f27c5e2032322c87c33864571461b7936b25a81a584a78962f89 WatchSource:0}: Error finding container a47f1768b776f27c5e2032322c87c33864571461b7936b25a81a584a78962f89: Status 404 returned error can't find the container with id a47f1768b776f27c5e2032322c87c33864571461b7936b25a81a584a78962f89
May 25 11:28:37.567106 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.567013    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d1e61e-0dfa-41bd-bf39-4387253f6b00-cert\") pod \"odh-model-controller-696fc77849-f6h99\" (UID: \"65d1e61e-0dfa-41bd-bf39-4387253f6b00\") " pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:37.569457 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.569432    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d1e61e-0dfa-41bd-bf39-4387253f6b00-cert\") pod \"odh-model-controller-696fc77849-f6h99\" (UID: \"65d1e61e-0dfa-41bd-bf39-4387253f6b00\") " pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:37.781887 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.781535    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:37.940712 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:37.940653    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-f6h99"]
May 25 11:28:37.942632 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:28:37.942591    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d1e61e_0dfa_41bd_bf39_4387253f6b00.slice/crio-164206d5fb17c264058db43f6f1b94a08872ca1d838fdca3188836a186e9aa04 WatchSource:0}: Error finding container 164206d5fb17c264058db43f6f1b94a08872ca1d838fdca3188836a186e9aa04: Status 404 returned error can't find the container with id 164206d5fb17c264058db43f6f1b94a08872ca1d838fdca3188836a186e9aa04
May 25 11:28:38.040205 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:38.040141    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-f6h99" event={"ID":"65d1e61e-0dfa-41bd-bf39-4387253f6b00","Type":"ContainerStarted","Data":"164206d5fb17c264058db43f6f1b94a08872ca1d838fdca3188836a186e9aa04"}
May 25 11:28:38.041503 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:38.041469    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-t5pkt" event={"ID":"0a34b29f-05db-4110-ad8e-d49951a1aa73","Type":"ContainerStarted","Data":"a47f1768b776f27c5e2032322c87c33864571461b7936b25a81a584a78962f89"}
May 25 11:28:41.056179 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:41.056133    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-f6h99" event={"ID":"65d1e61e-0dfa-41bd-bf39-4387253f6b00","Type":"ContainerStarted","Data":"d3db99b043432c52e262f32c70e6c9c03c929af94e73c97ce04a42384fcd4a57"}
May 25 11:28:41.056652 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:41.056210    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:41.057600 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:41.057568    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-t5pkt" event={"ID":"0a34b29f-05db-4110-ad8e-d49951a1aa73","Type":"ContainerStarted","Data":"88a32d245acda3fe106b8be983f72275dcaf0cd51550b065aff34851758c588b"}
May 25 11:28:41.057712 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:41.057663    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:41.075829 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:41.075781    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-f6h99" podStartSLOduration=2.13879151 podStartE2EDuration="5.075765762s" podCreationTimestamp="2026-05-25 11:28:36 +0000 UTC" firstStartedPulling="2026-05-25 11:28:37.94459396 +0000 UTC m=+676.699570740" lastFinishedPulling="2026-05-25 11:28:40.881568205 +0000 UTC m=+679.636544992" observedRunningTime="2026-05-25 11:28:41.073947579 +0000 UTC m=+679.828924401" watchObservedRunningTime="2026-05-25 11:28:41.075765762 +0000 UTC m=+679.830742558"
May 25 11:28:41.091370 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:41.091301    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-t5pkt" podStartSLOduration=1.5042536370000001 podStartE2EDuration="5.091279965s" podCreationTimestamp="2026-05-25 11:28:36 +0000 UTC" firstStartedPulling="2026-05-25 11:28:37.295481049 +0000 UTC m=+676.050457823" lastFinishedPulling="2026-05-25 11:28:40.882507375 +0000 UTC m=+679.637484151" observedRunningTime="2026-05-25 11:28:41.090143955 +0000 UTC m=+679.845120763" watchObservedRunningTime="2026-05-25 11:28:41.091279965 +0000 UTC m=+679.846256762"
May 25 11:28:52.064341 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:52.064309    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-f6h99"
May 25 11:28:52.066067 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:52.066050    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-t5pkt"
May 25 11:28:52.893071 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:52.893036    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rvn65"]
May 25 11:28:52.896089 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:52.896072    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvn65"
May 25 11:28:52.904057 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:52.904030    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rvn65"]
May 25 11:28:52.995718 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:52.995683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvhn\" (UniqueName: \"kubernetes.io/projected/cdde9eb4-7d47-43f6-9cf5-da674cbc0959-kube-api-access-9wvhn\") pod \"s3-init-rvn65\" (UID: \"cdde9eb4-7d47-43f6-9cf5-da674cbc0959\") " pod="kserve/s3-init-rvn65"
May 25 11:28:53.096252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:53.096214    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvhn\" (UniqueName: \"kubernetes.io/projected/cdde9eb4-7d47-43f6-9cf5-da674cbc0959-kube-api-access-9wvhn\") pod \"s3-init-rvn65\" (UID: \"cdde9eb4-7d47-43f6-9cf5-da674cbc0959\") " pod="kserve/s3-init-rvn65"
May 25 11:28:53.104373 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:53.104340    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvhn\" (UniqueName: \"kubernetes.io/projected/cdde9eb4-7d47-43f6-9cf5-da674cbc0959-kube-api-access-9wvhn\") pod \"s3-init-rvn65\" (UID: \"cdde9eb4-7d47-43f6-9cf5-da674cbc0959\") " pod="kserve/s3-init-rvn65"
May 25 11:28:53.205404 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:53.205317    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvn65"
May 25 11:28:53.338473 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:53.338446    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rvn65"]
May 25 11:28:53.339692 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:28:53.339670    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdde9eb4_7d47_43f6_9cf5_da674cbc0959.slice/crio-647a877888c57a9b749a7ed09f83aaf7bef88bb90dd97e9f3f11eff1818dd141 WatchSource:0}: Error finding container 647a877888c57a9b749a7ed09f83aaf7bef88bb90dd97e9f3f11eff1818dd141: Status 404 returned error can't find the container with id 647a877888c57a9b749a7ed09f83aaf7bef88bb90dd97e9f3f11eff1818dd141
May 25 11:28:54.107214 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:54.107171    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvn65" event={"ID":"cdde9eb4-7d47-43f6-9cf5-da674cbc0959","Type":"ContainerStarted","Data":"647a877888c57a9b749a7ed09f83aaf7bef88bb90dd97e9f3f11eff1818dd141"}
May 25 11:28:58.128602 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:58.128565    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvn65" event={"ID":"cdde9eb4-7d47-43f6-9cf5-da674cbc0959","Type":"ContainerStarted","Data":"8f58f161b6d8da4072a746f7319a9666fde250f488ff9e90a8e81d274cc41cfb"}
May 25 11:28:58.145424 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:28:58.145376    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-rvn65" podStartSLOduration=1.7744364419999998 podStartE2EDuration="6.145360426s" podCreationTimestamp="2026-05-25 11:28:52 +0000 UTC" firstStartedPulling="2026-05-25 11:28:53.34159098 +0000 UTC m=+692.096567754" lastFinishedPulling="2026-05-25 11:28:57.712514951 +0000 UTC m=+696.467491738" observedRunningTime="2026-05-25 11:28:58.143553926 +0000 UTC m=+696.898530716" watchObservedRunningTime="2026-05-25 11:28:58.145360426 +0000 UTC m=+696.900337267"
May 25 11:29:01.142493 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:01.142461    2566 generic.go:358] "Generic (PLEG): container finished" podID="cdde9eb4-7d47-43f6-9cf5-da674cbc0959" containerID="8f58f161b6d8da4072a746f7319a9666fde250f488ff9e90a8e81d274cc41cfb" exitCode=0
May 25 11:29:01.142866 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:01.142554    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvn65" event={"ID":"cdde9eb4-7d47-43f6-9cf5-da674cbc0959","Type":"ContainerDied","Data":"8f58f161b6d8da4072a746f7319a9666fde250f488ff9e90a8e81d274cc41cfb"}
May 25 11:29:02.284232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:02.284205    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvn65"
May 25 11:29:02.376819 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:02.376787    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wvhn\" (UniqueName: \"kubernetes.io/projected/cdde9eb4-7d47-43f6-9cf5-da674cbc0959-kube-api-access-9wvhn\") pod \"cdde9eb4-7d47-43f6-9cf5-da674cbc0959\" (UID: \"cdde9eb4-7d47-43f6-9cf5-da674cbc0959\") "
May 25 11:29:02.378790 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:02.378767    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdde9eb4-7d47-43f6-9cf5-da674cbc0959-kube-api-access-9wvhn" (OuterVolumeSpecName: "kube-api-access-9wvhn") pod "cdde9eb4-7d47-43f6-9cf5-da674cbc0959" (UID: "cdde9eb4-7d47-43f6-9cf5-da674cbc0959"). InnerVolumeSpecName "kube-api-access-9wvhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:29:02.478265 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:02.478161    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvhn\" (UniqueName: \"kubernetes.io/projected/cdde9eb4-7d47-43f6-9cf5-da674cbc0959-kube-api-access-9wvhn\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:29:03.151577 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:03.151528    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rvn65"
May 25 11:29:03.151577 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:03.151535    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rvn65" event={"ID":"cdde9eb4-7d47-43f6-9cf5-da674cbc0959","Type":"ContainerDied","Data":"647a877888c57a9b749a7ed09f83aaf7bef88bb90dd97e9f3f11eff1818dd141"}
May 25 11:29:03.151577 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:03.151579    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647a877888c57a9b749a7ed09f83aaf7bef88bb90dd97e9f3f11eff1818dd141"
May 25 11:29:13.337713 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.337621    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"]
May 25 11:29:13.338566 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.338023    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdde9eb4-7d47-43f6-9cf5-da674cbc0959" containerName="s3-init"
May 25 11:29:13.338566 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.338042    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdde9eb4-7d47-43f6-9cf5-da674cbc0959" containerName="s3-init"
May 25 11:29:13.338566 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.338107    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="cdde9eb4-7d47-43f6-9cf5-da674cbc0959" containerName="s3-init"
May 25 11:29:13.345140 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.345116    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.347739 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.347713    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-496r2\""
May 25 11:29:13.347999 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.347981    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
May 25 11:29:13.348131 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.348112    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:29:13.350173 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.350137    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:29:13.352822 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.352799    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"]
May 25 11:29:13.474968 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.474935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.474968 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.474971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.475201 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.474991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hwf\" (UniqueName: \"kubernetes.io/projected/0a16e53e-5908-4378-bf39-c8858e3a82c9-kube-api-access-79hwf\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.475201 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.475122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.475201 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.475183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.475308 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.475214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.475308 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.475253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.475308 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.475285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a16e53e-5908-4378-bf39-c8858e3a82c9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.475415 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.475321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576077 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576038    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576077 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576077    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576203    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576248    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a16e53e-5908-4378-bf39-c8858e3a82c9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576409 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576371    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576455 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576423    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576519 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576494    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576596 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576517    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576596 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576573    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-79hwf\" (UniqueName: \"kubernetes.io/projected/0a16e53e-5908-4378-bf39-c8858e3a82c9-kube-api-access-79hwf\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576695 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576592    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576695 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576627    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576837 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576816    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.576891 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.576870    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.577071 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.577041    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a16e53e-5908-4378-bf39-c8858e3a82c9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.578471 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.578448    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.579103 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.579084    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.584181 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.584157    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a16e53e-5908-4378-bf39-c8858e3a82c9-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.584421 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.584402    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hwf\" (UniqueName: \"kubernetes.io/projected/0a16e53e-5908-4378-bf39-c8858e3a82c9-kube-api-access-79hwf\") pod \"router-gateway-1-openshift-default-7b9cf87876-ggcgj\" (UID: \"0a16e53e-5908-4378-bf39-c8858e3a82c9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.659482 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.659452    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:13.805524 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.805489    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"]
May 25 11:29:13.807571 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:29:13.807528    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a16e53e_5908_4378_bf39_c8858e3a82c9.slice/crio-062611ee5271573d239c5683fb822e5ec076daa48b98aaad797c9b74bfd77139 WatchSource:0}: Error finding container 062611ee5271573d239c5683fb822e5ec076daa48b98aaad797c9b74bfd77139: Status 404 returned error can't find the container with id 062611ee5271573d239c5683fb822e5ec076daa48b98aaad797c9b74bfd77139
May 25 11:29:13.809667 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.809633    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:29:13.809751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.809698    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:29:13.809751 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:13.809732    2566 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 25 11:29:14.193182 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:14.193145    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj" event={"ID":"0a16e53e-5908-4378-bf39-c8858e3a82c9","Type":"ContainerStarted","Data":"da305f16a24667cbdd5ebe93f91c6f09d8a718cfffc4d3964c9a786837fa59ff"}
May 25 11:29:14.193363 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:14.193191    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj" event={"ID":"0a16e53e-5908-4378-bf39-c8858e3a82c9","Type":"ContainerStarted","Data":"062611ee5271573d239c5683fb822e5ec076daa48b98aaad797c9b74bfd77139"}
May 25 11:29:14.211793 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:14.211732    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj" podStartSLOduration=1.211711929 podStartE2EDuration="1.211711929s" podCreationTimestamp="2026-05-25 11:29:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:29:14.210977381 +0000 UTC m=+712.965954178" watchObservedRunningTime="2026-05-25 11:29:14.211711929 +0000 UTC m=+712.966688726"
May 25 11:29:14.660450 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:14.660409    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:14.666215 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:14.666187    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:15.196843 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:15.196812    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:15.197857 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:15.197839    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-ggcgj"
May 25 11:29:20.902486 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:20.902449    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"]
May 25 11:29:20.905188 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:20.905167    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:20.907883 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:20.907861    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 25 11:29:20.908002 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:20.907979    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5cxfn\""
May 25 11:29:20.908061 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:20.908007    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-czmsj\""
May 25 11:29:20.919150 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:20.919112    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"]
May 25 11:29:21.046570 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.046503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt496\" (UniqueName: \"kubernetes.io/projected/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kube-api-access-dt496\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.046759 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.046575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.046759 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.046629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.046759 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.046657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.046759 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.046693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.046938 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.046794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.147707 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.147667    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.147707 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.147705    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.147946 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.147732    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.147946 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.147782    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.147946 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.147813    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dt496\" (UniqueName: \"kubernetes.io/projected/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kube-api-access-dt496\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.147946 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.147830    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.148138 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.148096    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.148192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.148153    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.148192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.148169    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.148260 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.148193    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.150318 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.150300    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.156862 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.156772    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt496\" (UniqueName: \"kubernetes.io/projected/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kube-api-access-dt496\") pod \"scheduler-inline-config-test-kserve-router-scheduler-779d968kvb\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.214120 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.214086    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:21.361923 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:21.361897    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"]
May 25 11:29:21.363863 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:29:21.363835    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8662dbf_dfc4_43c2_93fe_dd0a53c54b41.slice/crio-e478df580afd6e15b8950e6814e90aa8f58bfbcf22a51fa615d266ced995a05c WatchSource:0}: Error finding container e478df580afd6e15b8950e6814e90aa8f58bfbcf22a51fa615d266ced995a05c: Status 404 returned error can't find the container with id e478df580afd6e15b8950e6814e90aa8f58bfbcf22a51fa615d266ced995a05c
May 25 11:29:22.243565 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:22.242817    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerStarted","Data":"e478df580afd6e15b8950e6814e90aa8f58bfbcf22a51fa615d266ced995a05c"}
May 25 11:29:25.261234 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:25.261195    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerStarted","Data":"542b81e4b8877f188f0a894177040dbc81dd1187836c0271fc4601bf63971408"}
May 25 11:29:26.266072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:26.266035    2566 generic.go:358] "Generic (PLEG): container finished" podID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerID="542b81e4b8877f188f0a894177040dbc81dd1187836c0271fc4601bf63971408" exitCode=0
May 25 11:29:26.266451 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:26.266106    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerDied","Data":"542b81e4b8877f188f0a894177040dbc81dd1187836c0271fc4601bf63971408"}
May 25 11:29:28.279460 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:28.279401    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerStarted","Data":"a524d0c5e155e428f105aa208d4f479a03027b06b0b4369bd42314c8337e24eb"}
May 25 11:29:58.401944 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:58.401901    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerStarted","Data":"f4b9373d5a210ecd9742c970ef33fef5b5867fd579cd203774da4ce86abf9a93"}
May 25 11:29:58.402420 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:58.402041    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:58.404577 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:58.404554    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:29:58.424873 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:29:58.424822    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" podStartSLOduration=1.71688389 podStartE2EDuration="38.424809012s" podCreationTimestamp="2026-05-25 11:29:20 +0000 UTC" firstStartedPulling="2026-05-25 11:29:21.3658176 +0000 UTC m=+720.120794374" lastFinishedPulling="2026-05-25 11:29:58.07374272 +0000 UTC m=+756.828719496" observedRunningTime="2026-05-25 11:29:58.421096399 +0000 UTC m=+757.176073200" watchObservedRunningTime="2026-05-25 11:29:58.424809012 +0000 UTC m=+757.179785821"
May 25 11:30:01.214726 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:01.214692    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:30:01.214726 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:01.214733    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:30:11.216418 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:11.216385    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:30:11.217853 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:11.217830    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:30:39.018864 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.018764    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"]
May 25 11:30:39.039943 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.039912    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"]
May 25 11:30:39.040103 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.040066    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.042642 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.042614    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-r54lg\""
May 25 11:30:39.042772 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.042711    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 25 11:30:39.141946 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.141911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.142112 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.141959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.142112 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.141980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9xdb\" (UniqueName: \"kubernetes.io/projected/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kube-api-access-l9xdb\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.142112 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.142063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.142218 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.142109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.142218 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.142162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243319 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243282    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243498 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243330    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243498 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243367    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243498 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243399    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243498 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243426    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9xdb\" (UniqueName: \"kubernetes.io/projected/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kube-api-access-l9xdb\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243498 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243475    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243806 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243758    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243916 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243803    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243916 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243821    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.243916 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.243849    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.246027 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.245996    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.252938 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.252910    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9xdb\" (UniqueName: \"kubernetes.io/projected/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kube-api-access-l9xdb\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.349691 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.349655    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:39.490106 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.490081    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"]
May 25 11:30:39.491718 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:30:39.491687    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e317d1_1e67_475e_bbf7_ddaba1f4c6a2.slice/crio-c1047dddb91d65b805a7cd1d95d8aa3584f3406f8d420ccd718ee74c618821b2 WatchSource:0}: Error finding container c1047dddb91d65b805a7cd1d95d8aa3584f3406f8d420ccd718ee74c618821b2: Status 404 returned error can't find the container with id c1047dddb91d65b805a7cd1d95d8aa3584f3406f8d420ccd718ee74c618821b2
May 25 11:30:39.549736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.549703    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerStarted","Data":"c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7"}
May 25 11:30:39.549840 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:39.549748    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerStarted","Data":"c1047dddb91d65b805a7cd1d95d8aa3584f3406f8d420ccd718ee74c618821b2"}
May 25 11:30:40.554502 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:40.554415    2566 generic.go:358] "Generic (PLEG): container finished" podID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerID="c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7" exitCode=0
May 25 11:30:40.554879 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:40.554497    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerDied","Data":"c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7"}
May 25 11:30:41.561077 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:41.561041    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerStarted","Data":"7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11"}
May 25 11:30:41.561077 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:41.561083    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerStarted","Data":"78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a"}
May 25 11:30:41.561499 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:41.561115    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:41.583007 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:41.582959    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" podStartSLOduration=3.582941721 podStartE2EDuration="3.582941721s" podCreationTimestamp="2026-05-25 11:30:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:30:41.581217731 +0000 UTC m=+800.336194577" watchObservedRunningTime="2026-05-25 11:30:41.582941721 +0000 UTC m=+800.337918516"
May 25 11:30:49.350168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:49.350119    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:49.350168 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:49.350171    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:49.352877 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:49.352844    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:30:49.595782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:30:49.595753    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:31:10.599648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:31:10.599616    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:32:55.489920 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:55.489881    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"]
May 25 11:32:55.492461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:55.490973    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="main" containerID="cri-o://78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a" gracePeriod=30
May 25 11:32:55.492461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:55.491001    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="tokenizer" containerID="cri-o://7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11" gracePeriod=30
May 25 11:32:56.060498 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.060461    2566 generic.go:358] "Generic (PLEG): container finished" podID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerID="78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a" exitCode=0
May 25 11:32:56.060695 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.060534    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerDied","Data":"78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a"}
May 25 11:32:56.744741 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.744719    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:32:56.905019 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.904981    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9xdb\" (UniqueName: \"kubernetes.io/projected/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kube-api-access-l9xdb\") pod \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") "
May 25 11:32:56.905200 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905035    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-uds\") pod \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") "
May 25 11:32:56.905200 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905060    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kserve-provision-location\") pod \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") "
May 25 11:32:56.905200 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905101    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-tmp\") pod \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") "
May 25 11:32:56.905200 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905131    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tls-certs\") pod \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") "
May 25 11:32:56.905200 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905175    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-cache\") pod \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\" (UID: \"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2\") "
May 25 11:32:56.905446 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905400    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" (UID: "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:32:56.905525 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905489    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" (UID: "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:32:56.905609 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905498    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" (UID: "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:32:56.905877 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.905856    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" (UID: "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:32:56.907306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.907285    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" (UID: "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:32:56.907527 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:56.907510    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kube-api-access-l9xdb" (OuterVolumeSpecName: "kube-api-access-l9xdb") pod "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" (UID: "15e317d1-1e67-475e-bbf7-ddaba1f4c6a2"). InnerVolumeSpecName "kube-api-access-l9xdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:32:57.006281 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.006243    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:32:57.006281 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.006275    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9xdb\" (UniqueName: \"kubernetes.io/projected/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kube-api-access-l9xdb\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:32:57.006281 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.006286    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:32:57.006509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.006296    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:32:57.006509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.006305    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:32:57.006509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.006314    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:32:57.066121 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.066082    2566 generic.go:358] "Generic (PLEG): container finished" podID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerID="7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11" exitCode=0
May 25 11:32:57.066301 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.066140    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerDied","Data":"7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11"}
May 25 11:32:57.066301 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.066161    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"
May 25 11:32:57.066301 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.066187    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr" event={"ID":"15e317d1-1e67-475e-bbf7-ddaba1f4c6a2","Type":"ContainerDied","Data":"c1047dddb91d65b805a7cd1d95d8aa3584f3406f8d420ccd718ee74c618821b2"}
May 25 11:32:57.066301 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.066208    2566 scope.go:117] "RemoveContainer" containerID="7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11"
May 25 11:32:57.075021 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.075002    2566 scope.go:117] "RemoveContainer" containerID="78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a"
May 25 11:32:57.082508 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.082489    2566 scope.go:117] "RemoveContainer" containerID="c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7"
May 25 11:32:57.089634 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.089607    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"]
May 25 11:32:57.090677 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.090658    2566 scope.go:117] "RemoveContainer" containerID="7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11"
May 25 11:32:57.091196 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:32:57.091173    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11\": container with ID starting with 7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11 not found: ID does not exist" containerID="7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11"
May 25 11:32:57.091276 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.091204    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11"} err="failed to get container status \"7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11\": rpc error: code = NotFound desc = could not find container \"7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11\": container with ID starting with 7e3fd1f4042d3d0a93e5016734b23bba38e85063d3ee89fdc8d1638fe4ea1a11 not found: ID does not exist"
May 25 11:32:57.091276 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.091232    2566 scope.go:117] "RemoveContainer" containerID="78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a"
May 25 11:32:57.091561 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:32:57.091518    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a\": container with ID starting with 78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a not found: ID does not exist" containerID="78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a"
May 25 11:32:57.091641 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.091567    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a"} err="failed to get container status \"78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a\": rpc error: code = NotFound desc = could not find container \"78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a\": container with ID starting with 78a102ad3a4d1cc19a880c9af7e66826fef32f8fddf35f64ec237f3d1e828c3a not found: ID does not exist"
May 25 11:32:57.091641 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.091593    2566 scope.go:117] "RemoveContainer" containerID="c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7"
May 25 11:32:57.091907 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:32:57.091873    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7\": container with ID starting with c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7 not found: ID does not exist" containerID="c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7"
May 25 11:32:57.091958 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.091907    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7"} err="failed to get container status \"c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7\": rpc error: code = NotFound desc = could not find container \"c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7\": container with ID starting with c8a023b1d2c89e521c89b85937bacd958777a1b3d97f8bb6ca4dcd8654d012e7 not found: ID does not exist"
May 25 11:32:57.093038 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.093020    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schevcwqr"]
May 25 11:32:57.841038 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:32:57.840997    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" path="/var/lib/kubelet/pods/15e317d1-1e67-475e-bbf7-ddaba1f4c6a2/volumes"
May 25 11:33:05.764252 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764199    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"]
May 25 11:33:05.764785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764760    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="tokenizer"
May 25 11:33:05.764785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764781    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="tokenizer"
May 25 11:33:05.764954 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764797    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="main"
May 25 11:33:05.764954 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764803    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="main"
May 25 11:33:05.764954 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764824    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="storage-initializer"
May 25 11:33:05.764954 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764834    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="storage-initializer"
May 25 11:33:05.764954 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764893    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="tokenizer"
May 25 11:33:05.764954 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.764907    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15e317d1-1e67-475e-bbf7-ddaba1f4c6a2" containerName="main"
May 25 11:33:05.768232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.768213    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.770585 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.770564    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-tdlm2\""
May 25 11:33:05.770689 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.770606    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 25 11:33:05.782683 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.782661    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"]
May 25 11:33:05.883710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.883673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.883710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.883717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.883982 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.883800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.883982 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.883841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwg4\" (UniqueName: \"kubernetes.io/projected/8dba4bda-bdc9-4beb-a361-98b51153888c-kube-api-access-pqwg4\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.883982 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.883896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dba4bda-bdc9-4beb-a361-98b51153888c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.883982 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.883935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.984532 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984498    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.984736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984591    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.984736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984616    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.984736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984642    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.984736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984661    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwg4\" (UniqueName: \"kubernetes.io/projected/8dba4bda-bdc9-4beb-a361-98b51153888c-kube-api-access-pqwg4\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.984736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984697    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dba4bda-bdc9-4beb-a361-98b51153888c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.985009 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984941    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.985009 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.984990    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.985104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.985041    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.985104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.985080    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.987228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.987206    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dba4bda-bdc9-4beb-a361-98b51153888c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:05.993253 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:05.993233    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwg4\" (UniqueName: \"kubernetes.io/projected/8dba4bda-bdc9-4beb-a361-98b51153888c-kube-api-access-pqwg4\") pod \"custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:06.078294 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:06.078205    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:06.216225 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:06.216198    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"]
May 25 11:33:06.218106 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:33:06.218079    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dba4bda_bdc9_4beb_a361_98b51153888c.slice/crio-b5c54692321ce1e3550c20e6ce539ce7031c0d621975406bcb9a89460eace6dd WatchSource:0}: Error finding container b5c54692321ce1e3550c20e6ce539ce7031c0d621975406bcb9a89460eace6dd: Status 404 returned error can't find the container with id b5c54692321ce1e3550c20e6ce539ce7031c0d621975406bcb9a89460eace6dd
May 25 11:33:06.220382 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:06.220365    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:33:07.106441 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:07.106400    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerStarted","Data":"897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e"}
May 25 11:33:07.106850 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:07.106448    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerStarted","Data":"b5c54692321ce1e3550c20e6ce539ce7031c0d621975406bcb9a89460eace6dd"}
May 25 11:33:08.111102 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:08.111063    2566 generic.go:358] "Generic (PLEG): container finished" podID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerID="897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e" exitCode=0
May 25 11:33:08.111507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:08.111145    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerDied","Data":"897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e"}
May 25 11:33:09.116784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:09.116749    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerStarted","Data":"5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2"}
May 25 11:33:09.116784 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:09.116785    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerStarted","Data":"753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d"}
May 25 11:33:09.117242 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:09.116905    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:09.141390 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:09.141337    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" podStartSLOduration=4.141319479 podStartE2EDuration="4.141319479s" podCreationTimestamp="2026-05-25 11:33:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:33:09.139226709 +0000 UTC m=+947.894203506" watchObservedRunningTime="2026-05-25 11:33:09.141319479 +0000 UTC m=+947.896296274"
May 25 11:33:16.078858 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:16.078815    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:16.079313 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:16.078883    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:16.081733 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:16.081708    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:16.145261 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:16.145213    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:33:38.152589 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:33:38.152500    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:35:21.542600 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:21.542501    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"]
May 25 11:35:21.543037 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:21.542856    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="main" containerID="cri-o://753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d" gracePeriod=30
May 25 11:35:21.543037 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:21.542881    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="tokenizer" containerID="cri-o://5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2" gracePeriod=30
May 25 11:35:22.610719 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.610685    2566 generic.go:358] "Generic (PLEG): container finished" podID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerID="753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d" exitCode=0
May 25 11:35:22.611090 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.610753    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerDied","Data":"753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d"}
May 25 11:35:22.894906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.894874    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:35:22.971782 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.971749    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-kserve-provision-location\") pod \"8dba4bda-bdc9-4beb-a361-98b51153888c\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") "
May 25 11:35:22.971957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.971793    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dba4bda-bdc9-4beb-a361-98b51153888c-tls-certs\") pod \"8dba4bda-bdc9-4beb-a361-98b51153888c\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") "
May 25 11:35:22.971957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.971824    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-cache\") pod \"8dba4bda-bdc9-4beb-a361-98b51153888c\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") "
May 25 11:35:22.971957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.971861    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-tmp\") pod \"8dba4bda-bdc9-4beb-a361-98b51153888c\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") "
May 25 11:35:22.971957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.971879    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-uds\") pod \"8dba4bda-bdc9-4beb-a361-98b51153888c\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") "
May 25 11:35:22.972150 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.971966    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqwg4\" (UniqueName: \"kubernetes.io/projected/8dba4bda-bdc9-4beb-a361-98b51153888c-kube-api-access-pqwg4\") pod \"8dba4bda-bdc9-4beb-a361-98b51153888c\" (UID: \"8dba4bda-bdc9-4beb-a361-98b51153888c\") "
May 25 11:35:22.972209 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.972155    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "8dba4bda-bdc9-4beb-a361-98b51153888c" (UID: "8dba4bda-bdc9-4beb-a361-98b51153888c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:22.972263 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.972244    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8dba4bda-bdc9-4beb-a361-98b51153888c" (UID: "8dba4bda-bdc9-4beb-a361-98b51153888c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:22.972308 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.972253    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "8dba4bda-bdc9-4beb-a361-98b51153888c" (UID: "8dba4bda-bdc9-4beb-a361-98b51153888c"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:22.972641 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.972619    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8dba4bda-bdc9-4beb-a361-98b51153888c" (UID: "8dba4bda-bdc9-4beb-a361-98b51153888c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:22.973997 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.973977    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dba4bda-bdc9-4beb-a361-98b51153888c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dba4bda-bdc9-4beb-a361-98b51153888c" (UID: "8dba4bda-bdc9-4beb-a361-98b51153888c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:35:22.974132 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:22.974114    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dba4bda-bdc9-4beb-a361-98b51153888c-kube-api-access-pqwg4" (OuterVolumeSpecName: "kube-api-access-pqwg4") pod "8dba4bda-bdc9-4beb-a361-98b51153888c" (UID: "8dba4bda-bdc9-4beb-a361-98b51153888c"). InnerVolumeSpecName "kube-api-access-pqwg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:35:23.073462 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.073424    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:35:23.073462 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.073455    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwg4\" (UniqueName: \"kubernetes.io/projected/8dba4bda-bdc9-4beb-a361-98b51153888c-kube-api-access-pqwg4\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:35:23.073462 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.073467    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:35:23.073735 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.073477    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dba4bda-bdc9-4beb-a361-98b51153888c-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:35:23.073735 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.073489    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:35:23.073735 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.073497    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dba4bda-bdc9-4beb-a361-98b51153888c-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:35:23.616195 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.616157    2566 generic.go:358] "Generic (PLEG): container finished" podID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerID="5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2" exitCode=0
May 25 11:35:23.616660 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.616232    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerDied","Data":"5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2"}
May 25 11:35:23.616660 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.616255    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"
May 25 11:35:23.616660 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.616273    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj" event={"ID":"8dba4bda-bdc9-4beb-a361-98b51153888c","Type":"ContainerDied","Data":"b5c54692321ce1e3550c20e6ce539ce7031c0d621975406bcb9a89460eace6dd"}
May 25 11:35:23.616660 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.616289    2566 scope.go:117] "RemoveContainer" containerID="5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2"
May 25 11:35:23.626863 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.626837    2566 scope.go:117] "RemoveContainer" containerID="753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d"
May 25 11:35:23.637779 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.637723    2566 scope.go:117] "RemoveContainer" containerID="897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e"
May 25 11:35:23.640513 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.640490    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"]
May 25 11:35:23.646091 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.646063    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5df6cd7bc2lxj"]
May 25 11:35:23.646842 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.646826    2566 scope.go:117] "RemoveContainer" containerID="5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2"
May 25 11:35:23.647126 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:35:23.647109    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2\": container with ID starting with 5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2 not found: ID does not exist" containerID="5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2"
May 25 11:35:23.647189 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.647144    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2"} err="failed to get container status \"5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2\": rpc error: code = NotFound desc = could not find container \"5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2\": container with ID starting with 5825fa1246e9c679a57fce0e26fd29387c2f8bab3fb2e07f7b14de4b698c35f2 not found: ID does not exist"
May 25 11:35:23.647189 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.647164    2566 scope.go:117] "RemoveContainer" containerID="753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d"
May 25 11:35:23.647434 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:35:23.647418    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d\": container with ID starting with 753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d not found: ID does not exist" containerID="753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d"
May 25 11:35:23.647486 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.647439    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d"} err="failed to get container status \"753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d\": rpc error: code = NotFound desc = could not find container \"753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d\": container with ID starting with 753fcda650ded11ef3aac643a675723d14f738fd98bbb2a9b8aa5510a0e3752d not found: ID does not exist"
May 25 11:35:23.647486 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.647453    2566 scope.go:117] "RemoveContainer" containerID="897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e"
May 25 11:35:23.647789 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:35:23.647770    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e\": container with ID starting with 897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e not found: ID does not exist" containerID="897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e"
May 25 11:35:23.647833 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.647796    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e"} err="failed to get container status \"897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e\": rpc error: code = NotFound desc = could not find container \"897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e\": container with ID starting with 897055bfed0295e55ecc8d4585ba5d0dde033f636b5e8e63fd66168d7fc71d8e not found: ID does not exist"
May 25 11:35:23.838914 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:23.838870    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" path="/var/lib/kubelet/pods/8dba4bda-bdc9-4beb-a361-98b51153888c/volumes"
May 25 11:35:40.552332 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552298    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"]
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552661    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="storage-initializer"
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552673    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="storage-initializer"
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552689    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="main"
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552695    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="main"
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552701    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="tokenizer"
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552706    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="tokenizer"
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552763    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="tokenizer"
May 25 11:35:40.552826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.552773    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="8dba4bda-bdc9-4beb-a361-98b51153888c" containerName="main"
May 25 11:35:40.557755 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.557733    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.568404 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.568383    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 25 11:35:40.580010 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.579987    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-xdxpf\""
May 25 11:35:40.602150 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.602115    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"]
May 25 11:35:40.726579 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.726506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.726579 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.726582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.726847 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.726641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.726847 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.726726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq7tg\" (UniqueName: \"kubernetes.io/projected/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kube-api-access-nq7tg\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.726847 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.726754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.726847 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.726788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828029 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.827936    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7tg\" (UniqueName: \"kubernetes.io/projected/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kube-api-access-nq7tg\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828029 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.827981    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828029 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828012    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828040    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828066    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828124    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828511 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828489    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828595 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828513    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828595 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828562    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.828670 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.828628    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.830742 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.830722    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.849843 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.849806    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7tg\" (UniqueName: \"kubernetes.io/projected/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kube-api-access-nq7tg\") pod \"router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:40.867674 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:40.867632    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:41.048800 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:41.048766    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"]
May 25 11:35:41.049929 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:35:41.049904    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12786ff_4f89_4277_9e09_2b0199a5b7aa.slice/crio-7d72136e7fc49f79fc3682918a194391aff79ca161de71aa00764a44e21ad403 WatchSource:0}: Error finding container 7d72136e7fc49f79fc3682918a194391aff79ca161de71aa00764a44e21ad403: Status 404 returned error can't find the container with id 7d72136e7fc49f79fc3682918a194391aff79ca161de71aa00764a44e21ad403
May 25 11:35:41.688089 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:41.688056    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerStarted","Data":"75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030"}
May 25 11:35:41.688089 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:41.688093    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerStarted","Data":"7d72136e7fc49f79fc3682918a194391aff79ca161de71aa00764a44e21ad403"}
May 25 11:35:42.692932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:42.692897    2566 generic.go:358] "Generic (PLEG): container finished" podID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerID="75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030" exitCode=0
May 25 11:35:42.693322 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:42.692958    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerDied","Data":"75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030"}
May 25 11:35:43.699120 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:43.699084    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerStarted","Data":"41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793"}
May 25 11:35:43.699120 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:43.699125    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerStarted","Data":"6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4"}
May 25 11:35:43.699558 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:43.699219    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:43.740578 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:43.740474    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" podStartSLOduration=3.740453645 podStartE2EDuration="3.740453645s" podCreationTimestamp="2026-05-25 11:35:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:35:43.740342991 +0000 UTC m=+1102.495319800" watchObservedRunningTime="2026-05-25 11:35:43.740453645 +0000 UTC m=+1102.495430443"
May 25 11:35:50.868731 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:50.868684    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:50.868731 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:50.868742    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:50.871664 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:50.871637    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:35:51.729099 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:35:51.729071    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:36:12.733365 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:36:12.733333    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:37:36.414950 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:36.414909    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"]
May 25 11:37:36.415565 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:36.415197    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="main" containerID="cri-o://6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4" gracePeriod=30
May 25 11:37:36.415565 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:36.415264    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="tokenizer" containerID="cri-o://41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793" gracePeriod=30
May 25 11:37:37.120932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.120889    2566 generic.go:358] "Generic (PLEG): container finished" podID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerID="6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4" exitCode=0
May 25 11:37:37.121104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.120954    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerDied","Data":"6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4"}
May 25 11:37:37.673587 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.673565    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:37:37.743122 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743088    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tls-certs\") pod \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") "
May 25 11:37:37.743293 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743165    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kserve-provision-location\") pod \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") "
May 25 11:37:37.743293 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743194    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-uds\") pod \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") "
May 25 11:37:37.743293 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743221    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq7tg\" (UniqueName: \"kubernetes.io/projected/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kube-api-access-nq7tg\") pod \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") "
May 25 11:37:37.743293 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743252    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-tmp\") pod \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") "
May 25 11:37:37.743516 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743324    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-cache\") pod \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\" (UID: \"c12786ff-4f89-4277-9e09-2b0199a5b7aa\") "
May 25 11:37:37.743516 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743483    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c12786ff-4f89-4277-9e09-2b0199a5b7aa" (UID: "c12786ff-4f89-4277-9e09-2b0199a5b7aa"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:37.743660 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743587    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:37:37.743721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743688    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "c12786ff-4f89-4277-9e09-2b0199a5b7aa" (UID: "c12786ff-4f89-4277-9e09-2b0199a5b7aa"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:37.743721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743704    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "c12786ff-4f89-4277-9e09-2b0199a5b7aa" (UID: "c12786ff-4f89-4277-9e09-2b0199a5b7aa"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:37.743996 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.743975    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c12786ff-4f89-4277-9e09-2b0199a5b7aa" (UID: "c12786ff-4f89-4277-9e09-2b0199a5b7aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:37.745228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.745209    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c12786ff-4f89-4277-9e09-2b0199a5b7aa" (UID: "c12786ff-4f89-4277-9e09-2b0199a5b7aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:37:37.745352 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.745327    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kube-api-access-nq7tg" (OuterVolumeSpecName: "kube-api-access-nq7tg") pod "c12786ff-4f89-4277-9e09-2b0199a5b7aa" (UID: "c12786ff-4f89-4277-9e09-2b0199a5b7aa"). InnerVolumeSpecName "kube-api-access-nq7tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:37:37.844878 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.844856    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:37:37.844960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.844882    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:37:37.844960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.844891    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12786ff-4f89-4277-9e09-2b0199a5b7aa-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:37:37.844960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.844903    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:37:37.844960 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:37.844912    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq7tg\" (UniqueName: \"kubernetes.io/projected/c12786ff-4f89-4277-9e09-2b0199a5b7aa-kube-api-access-nq7tg\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:37:38.126802 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.126702    2566 generic.go:358] "Generic (PLEG): container finished" podID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerID="41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793" exitCode=0
May 25 11:37:38.126802 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.126796    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"
May 25 11:37:38.127025 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.126788    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerDied","Data":"41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793"}
May 25 11:37:38.127025 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.126918    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw" event={"ID":"c12786ff-4f89-4277-9e09-2b0199a5b7aa","Type":"ContainerDied","Data":"7d72136e7fc49f79fc3682918a194391aff79ca161de71aa00764a44e21ad403"}
May 25 11:37:38.127025 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.126944    2566 scope.go:117] "RemoveContainer" containerID="41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793"
May 25 11:37:38.135754 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.135736    2566 scope.go:117] "RemoveContainer" containerID="6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4"
May 25 11:37:38.143257 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.143238    2566 scope.go:117] "RemoveContainer" containerID="75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030"
May 25 11:37:38.150496 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.150471    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"]
May 25 11:37:38.151250 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.151235    2566 scope.go:117] "RemoveContainer" containerID="41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793"
May 25 11:37:38.151504 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:37:38.151486    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793\": container with ID starting with 41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793 not found: ID does not exist" containerID="41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793"
May 25 11:37:38.151635 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.151518    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793"} err="failed to get container status \"41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793\": rpc error: code = NotFound desc = could not find container \"41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793\": container with ID starting with 41b0c018c989d8512eb3405b3c8f55dc54fcdf34459f8094cb4c2c5008c75793 not found: ID does not exist"
May 25 11:37:38.151635 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.151555    2566 scope.go:117] "RemoveContainer" containerID="6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4"
May 25 11:37:38.151795 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:37:38.151780    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4\": container with ID starting with 6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4 not found: ID does not exist" containerID="6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4"
May 25 11:37:38.151839 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.151799    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4"} err="failed to get container status \"6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4\": rpc error: code = NotFound desc = could not find container \"6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4\": container with ID starting with 6da6000476e8154e1bccf113df65c0950485827be173f969a117805c076682d4 not found: ID does not exist"
May 25 11:37:38.151839 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.151813    2566 scope.go:117] "RemoveContainer" containerID="75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030"
May 25 11:37:38.152005 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:37:38.151989    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030\": container with ID starting with 75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030 not found: ID does not exist" containerID="75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030"
May 25 11:37:38.152051 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.152010    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030"} err="failed to get container status \"75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030\": rpc error: code = NotFound desc = could not find container \"75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030\": container with ID starting with 75ac0a97e598c538d0aed319c6ad5b2a15bcc21f72ac6d87dfed311884705030 not found: ID does not exist"
May 25 11:37:38.157514 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:38.157485    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-677bfdd774-ncwfw"]
May 25 11:37:39.835700 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:39.835659    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" path="/var/lib/kubelet/pods/c12786ff-4f89-4277-9e09-2b0199a5b7aa/volumes"
May 25 11:37:40.981504 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981470    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"]
May 25 11:37:40.981909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981820    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="storage-initializer"
May 25 11:37:40.981909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981833    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="storage-initializer"
May 25 11:37:40.981909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981849    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="tokenizer"
May 25 11:37:40.981909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981855    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="tokenizer"
May 25 11:37:40.981909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981866    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="main"
May 25 11:37:40.981909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981871    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="main"
May 25 11:37:40.982109 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981919    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="tokenizer"
May 25 11:37:40.982109 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.981928    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="c12786ff-4f89-4277-9e09-2b0199a5b7aa" containerName="main"
May 25 11:37:40.986655 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.986634    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:40.989097 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.989076    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-ckm4d\""
May 25 11:37:40.989427 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:40.989410    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 25 11:37:41.006735 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.006706    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"]
May 25 11:37:41.071842 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.071803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.072036 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.071854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.072036 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.071887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.072036 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.071917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.072036 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.071951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhx8\" (UniqueName: \"kubernetes.io/projected/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kube-api-access-lqhx8\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.072036 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.072005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172412 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172366    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172639 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172424    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhx8\" (UniqueName: \"kubernetes.io/projected/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kube-api-access-lqhx8\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172639 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172455    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172639 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172478    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172639 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172516    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172864 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172661    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172922 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172884    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172975 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172926    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.172975 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172946    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.173039 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.172976    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.175119 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.175093    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.180835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.180801    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhx8\" (UniqueName: \"kubernetes.io/projected/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kube-api-access-lqhx8\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.296529 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.296429    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:41.442934 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:41.442834    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"]
May 25 11:37:41.444364 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:37:41.444333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25b5d7f_fcbb_4be6_98a4_607bef2c70da.slice/crio-133594b73ee095f60b4f03d5f3335ff1a1e80ee5141247eae6e5e363c02a735f WatchSource:0}: Error finding container 133594b73ee095f60b4f03d5f3335ff1a1e80ee5141247eae6e5e363c02a735f: Status 404 returned error can't find the container with id 133594b73ee095f60b4f03d5f3335ff1a1e80ee5141247eae6e5e363c02a735f
May 25 11:37:42.143917 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:42.143879    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerStarted","Data":"7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459"}
May 25 11:37:42.143917 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:42.143918    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerStarted","Data":"133594b73ee095f60b4f03d5f3335ff1a1e80ee5141247eae6e5e363c02a735f"}
May 25 11:37:43.148511 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:43.148476    2566 generic.go:358] "Generic (PLEG): container finished" podID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerID="7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459" exitCode=0
May 25 11:37:43.148903 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:43.148570    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerDied","Data":"7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459"}
May 25 11:37:44.154126 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:44.154091    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerStarted","Data":"691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7"}
May 25 11:37:44.154126 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:44.154126    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerStarted","Data":"6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f"}
May 25 11:37:44.154525 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:44.154222    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:44.179646 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:44.179590    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" podStartSLOduration=4.179574823 podStartE2EDuration="4.179574823s" podCreationTimestamp="2026-05-25 11:37:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:44.177814505 +0000 UTC m=+1222.932791342" watchObservedRunningTime="2026-05-25 11:37:44.179574823 +0000 UTC m=+1222.934551619"
May 25 11:37:51.296906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:51.296857    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:51.296906 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:51.296912    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:51.299621 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:51.299593    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:37:52.185187 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:37:52.185157    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:38:13.189044 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:38:13.188965    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:40:08.980467 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:08.980434    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"]
May 25 11:40:08.981051 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:08.980797    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="main" containerID="cri-o://6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f" gracePeriod=30
May 25 11:40:08.981051 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:08.980902    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="tokenizer" containerID="cri-o://691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7" gracePeriod=30
May 25 11:40:09.684264 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:09.684229    2566 generic.go:358] "Generic (PLEG): container finished" podID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerID="6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f" exitCode=0
May 25 11:40:09.684434 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:09.684294    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerDied","Data":"6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f"}
May 25 11:40:10.231853 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.231829    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:40:10.253596 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.253465    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqhx8\" (UniqueName: \"kubernetes.io/projected/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kube-api-access-lqhx8\") pod \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") "
May 25 11:40:10.253774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.253610    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-cache\") pod \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") "
May 25 11:40:10.253774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.253653    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tls-certs\") pod \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") "
May 25 11:40:10.253774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.253747    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-uds\") pod \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") "
May 25 11:40:10.253939 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.253784    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kserve-provision-location\") pod \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") "
May 25 11:40:10.253939 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.253806    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-tmp\") pod \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\" (UID: \"a25b5d7f-fcbb-4be6-98a4-607bef2c70da\") "
May 25 11:40:10.254224 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.254197    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a25b5d7f-fcbb-4be6-98a4-607bef2c70da" (UID: "a25b5d7f-fcbb-4be6-98a4-607bef2c70da"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:10.254327 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.254228    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a25b5d7f-fcbb-4be6-98a4-607bef2c70da" (UID: "a25b5d7f-fcbb-4be6-98a4-607bef2c70da"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:10.254392 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.254320    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "a25b5d7f-fcbb-4be6-98a4-607bef2c70da" (UID: "a25b5d7f-fcbb-4be6-98a4-607bef2c70da"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:10.255015 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.254891    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a25b5d7f-fcbb-4be6-98a4-607bef2c70da" (UID: "a25b5d7f-fcbb-4be6-98a4-607bef2c70da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:10.256449 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.256424    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kube-api-access-lqhx8" (OuterVolumeSpecName: "kube-api-access-lqhx8") pod "a25b5d7f-fcbb-4be6-98a4-607bef2c70da" (UID: "a25b5d7f-fcbb-4be6-98a4-607bef2c70da"). InnerVolumeSpecName "kube-api-access-lqhx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:40:10.256939 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.256913    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a25b5d7f-fcbb-4be6-98a4-607bef2c70da" (UID: "a25b5d7f-fcbb-4be6-98a4-607bef2c70da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:40:10.355384 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.355351    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:40:10.355384 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.355384    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:40:10.355615 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.355395    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:40:10.355615 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.355405    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqhx8\" (UniqueName: \"kubernetes.io/projected/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-kube-api-access-lqhx8\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:40:10.355615 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.355413    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:40:10.355615 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.355422    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a25b5d7f-fcbb-4be6-98a4-607bef2c70da-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:40:10.689836 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.689746    2566 generic.go:358] "Generic (PLEG): container finished" podID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerID="691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7" exitCode=0
May 25 11:40:10.689836 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.689818    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"
May 25 11:40:10.690059 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.689829    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerDied","Data":"691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7"}
May 25 11:40:10.690059 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.689873    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq" event={"ID":"a25b5d7f-fcbb-4be6-98a4-607bef2c70da","Type":"ContainerDied","Data":"133594b73ee095f60b4f03d5f3335ff1a1e80ee5141247eae6e5e363c02a735f"}
May 25 11:40:10.690059 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.689897    2566 scope.go:117] "RemoveContainer" containerID="691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7"
May 25 11:40:10.698781 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.698762    2566 scope.go:117] "RemoveContainer" containerID="6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f"
May 25 11:40:10.706305 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.706286    2566 scope.go:117] "RemoveContainer" containerID="7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459"
May 25 11:40:10.714436 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.714403    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"]
May 25 11:40:10.715915 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.715896    2566 scope.go:117] "RemoveContainer" containerID="691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7"
May 25 11:40:10.716294 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:40:10.716269    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7\": container with ID starting with 691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7 not found: ID does not exist" containerID="691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7"
May 25 11:40:10.716406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.716303    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7"} err="failed to get container status \"691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7\": rpc error: code = NotFound desc = could not find container \"691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7\": container with ID starting with 691ab194412d80b895e69e5df6a94a3581ba329eb0046090903722a385273ed7 not found: ID does not exist"
May 25 11:40:10.716406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.716328    2566 scope.go:117] "RemoveContainer" containerID="6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f"
May 25 11:40:10.716684 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:40:10.716658    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f\": container with ID starting with 6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f not found: ID does not exist" containerID="6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f"
May 25 11:40:10.716739 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.716695    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f"} err="failed to get container status \"6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f\": rpc error: code = NotFound desc = could not find container \"6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f\": container with ID starting with 6534fe2a25ef3d953168397e4ccab2463fef0e76b6374d0d6a8b2c2de924061f not found: ID does not exist"
May 25 11:40:10.716739 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.716712    2566 scope.go:117] "RemoveContainer" containerID="7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459"
May 25 11:40:10.716986 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:40:10.716971    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459\": container with ID starting with 7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459 not found: ID does not exist" containerID="7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459"
May 25 11:40:10.717047 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.716992    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459"} err="failed to get container status \"7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459\": rpc error: code = NotFound desc = could not find container \"7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459\": container with ID starting with 7d56a52d05a306fdd2077cbb7196e6081333366e6b56ee3931ba19088df20459 not found: ID does not exist"
May 25 11:40:10.717700 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:10.717680    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schev5qqq"]
May 25 11:40:11.835883 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:11.835846    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" path="/var/lib/kubelet/pods/a25b5d7f-fcbb-4be6-98a4-607bef2c70da/volumes"
May 25 11:40:23.791130 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791098    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"]
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791448    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="tokenizer"
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791462    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="tokenizer"
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791475    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="main"
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791482    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="main"
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791500    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="storage-initializer"
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791506    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="storage-initializer"
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791590    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="tokenizer"
May 25 11:40:23.791648 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.791598    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="a25b5d7f-fcbb-4be6-98a4-607bef2c70da" containerName="main"
May 25 11:40:23.796032 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.796012    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.800641 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.800616    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 25 11:40:23.800864 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.800841    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-xkfrr\""
May 25 11:40:23.809917 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.809892    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"]
May 25 11:40:23.873697 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.873661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.873866 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.873709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.873866 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.873773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9xf\" (UniqueName: \"kubernetes.io/projected/643b3354-7e20-4b44-af49-174ff81a728a-kube-api-access-nm9xf\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.873866 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.873818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.873866 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.873866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.874006 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.873886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3354-7e20-4b44-af49-174ff81a728a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975274 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975237    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975445 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975289    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9xf\" (UniqueName: \"kubernetes.io/projected/643b3354-7e20-4b44-af49-174ff81a728a-kube-api-access-nm9xf\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975445 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975317    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975445 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975352    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975445 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975370    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3354-7e20-4b44-af49-174ff81a728a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975669 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975550    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975725 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975702    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975771 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975743    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975833 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975812    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.975883 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.975868    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.977854 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.977831    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3354-7e20-4b44-af49-174ff81a728a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:23.984389 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:23.984362    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9xf\" (UniqueName: \"kubernetes.io/projected/643b3354-7e20-4b44-af49-174ff81a728a-kube-api-access-nm9xf\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:24.108326 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:24.108286    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:24.259795 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:40:24.259761    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643b3354_7e20_4b44_af49_174ff81a728a.slice/crio-caf39b4c915b3601a088d34443d59a33aaa24b763bf9bc9f58b1958070634f35 WatchSource:0}: Error finding container caf39b4c915b3601a088d34443d59a33aaa24b763bf9bc9f58b1958070634f35: Status 404 returned error can't find the container with id caf39b4c915b3601a088d34443d59a33aaa24b763bf9bc9f58b1958070634f35
May 25 11:40:24.260739 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:24.260717    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"]
May 25 11:40:24.261990 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:24.261970    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:40:24.742317 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:24.742274    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerStarted","Data":"00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d"}
May 25 11:40:24.742527 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:24.742334    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerStarted","Data":"caf39b4c915b3601a088d34443d59a33aaa24b763bf9bc9f58b1958070634f35"}
May 25 11:40:25.747455 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:25.747417    2566 generic.go:358] "Generic (PLEG): container finished" podID="643b3354-7e20-4b44-af49-174ff81a728a" containerID="00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d" exitCode=0
May 25 11:40:25.747967 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:25.747503    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerDied","Data":"00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d"}
May 25 11:40:26.752790 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:26.752756    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerStarted","Data":"f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a"}
May 25 11:40:26.753197 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:26.752794    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerStarted","Data":"b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276"}
May 25 11:40:26.753197 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:26.752924    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:26.776401 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:26.776349    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" podStartSLOduration=3.776331662 podStartE2EDuration="3.776331662s" podCreationTimestamp="2026-05-25 11:40:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:40:26.775615926 +0000 UTC m=+1385.530592723" watchObservedRunningTime="2026-05-25 11:40:26.776331662 +0000 UTC m=+1385.531308458"
May 25 11:40:34.109312 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:34.109275    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:34.109312 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:34.109324    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:34.112091 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:34.112065    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:34.784204 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:34.784170    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:40:55.788048 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:40:55.788021    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:42:51.556054 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:51.555960    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"]
May 25 11:42:51.556612 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:51.556376    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="main" containerID="cri-o://b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276" gracePeriod=30
May 25 11:42:51.556612 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:51.556416    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="tokenizer" containerID="cri-o://f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a" gracePeriod=30
May 25 11:42:52.280998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.280953    2566 generic.go:358] "Generic (PLEG): container finished" podID="643b3354-7e20-4b44-af49-174ff81a728a" containerID="b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276" exitCode=0
May 25 11:42:52.281176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.281024    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerDied","Data":"b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276"}
May 25 11:42:52.917655 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.917629    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:42:52.968895 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.968860    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-tmp\") pod \"643b3354-7e20-4b44-af49-174ff81a728a\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") "
May 25 11:42:52.968895 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.968898    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3354-7e20-4b44-af49-174ff81a728a-tls-certs\") pod \"643b3354-7e20-4b44-af49-174ff81a728a\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") "
May 25 11:42:52.969149 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.968955    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm9xf\" (UniqueName: \"kubernetes.io/projected/643b3354-7e20-4b44-af49-174ff81a728a-kube-api-access-nm9xf\") pod \"643b3354-7e20-4b44-af49-174ff81a728a\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") "
May 25 11:42:52.969149 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.968973    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-kserve-provision-location\") pod \"643b3354-7e20-4b44-af49-174ff81a728a\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") "
May 25 11:42:52.969149 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.969003    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-cache\") pod \"643b3354-7e20-4b44-af49-174ff81a728a\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") "
May 25 11:42:52.969149 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.969060    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-uds\") pod \"643b3354-7e20-4b44-af49-174ff81a728a\" (UID: \"643b3354-7e20-4b44-af49-174ff81a728a\") "
May 25 11:42:52.969361 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.969251    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "643b3354-7e20-4b44-af49-174ff81a728a" (UID: "643b3354-7e20-4b44-af49-174ff81a728a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:42:52.969361 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.969336    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "643b3354-7e20-4b44-af49-174ff81a728a" (UID: "643b3354-7e20-4b44-af49-174ff81a728a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:42:52.969452 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.969430    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "643b3354-7e20-4b44-af49-174ff81a728a" (UID: "643b3354-7e20-4b44-af49-174ff81a728a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:42:52.970299 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.970269    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "643b3354-7e20-4b44-af49-174ff81a728a" (UID: "643b3354-7e20-4b44-af49-174ff81a728a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:42:52.971201 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.971179    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643b3354-7e20-4b44-af49-174ff81a728a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "643b3354-7e20-4b44-af49-174ff81a728a" (UID: "643b3354-7e20-4b44-af49-174ff81a728a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:42:52.971411 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:52.971385    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643b3354-7e20-4b44-af49-174ff81a728a-kube-api-access-nm9xf" (OuterVolumeSpecName: "kube-api-access-nm9xf") pod "643b3354-7e20-4b44-af49-174ff81a728a" (UID: "643b3354-7e20-4b44-af49-174ff81a728a"). InnerVolumeSpecName "kube-api-access-nm9xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:42:53.070070 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.070012    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:42:53.070070 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.070063    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:42:53.070070 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.070074    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:42:53.070312 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.070083    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3354-7e20-4b44-af49-174ff81a728a-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:42:53.070312 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.070092    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9xf\" (UniqueName: \"kubernetes.io/projected/643b3354-7e20-4b44-af49-174ff81a728a-kube-api-access-nm9xf\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:42:53.070312 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.070103    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643b3354-7e20-4b44-af49-174ff81a728a-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:42:53.288072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.287981    2566 generic.go:358] "Generic (PLEG): container finished" podID="643b3354-7e20-4b44-af49-174ff81a728a" containerID="f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a" exitCode=0
May 25 11:42:53.288072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.288036    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerDied","Data":"f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a"}
May 25 11:42:53.288072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.288055    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"
May 25 11:42:53.288072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.288072    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg" event={"ID":"643b3354-7e20-4b44-af49-174ff81a728a","Type":"ContainerDied","Data":"caf39b4c915b3601a088d34443d59a33aaa24b763bf9bc9f58b1958070634f35"}
May 25 11:42:53.288386 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.288095    2566 scope.go:117] "RemoveContainer" containerID="f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a"
May 25 11:42:53.297378 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.297360    2566 scope.go:117] "RemoveContainer" containerID="b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276"
May 25 11:42:53.305166 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.305142    2566 scope.go:117] "RemoveContainer" containerID="00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d"
May 25 11:42:53.312493 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.312469    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"]
May 25 11:42:53.313754 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.313733    2566 scope.go:117] "RemoveContainer" containerID="f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a"
May 25 11:42:53.314055 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:42:53.314023    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a\": container with ID starting with f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a not found: ID does not exist" containerID="f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a"
May 25 11:42:53.314146 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.314066    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a"} err="failed to get container status \"f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a\": rpc error: code = NotFound desc = could not find container \"f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a\": container with ID starting with f80bd1f88409dfb303c248be922a169eba5077e36978f8069a07862326c7cc9a not found: ID does not exist"
May 25 11:42:53.314146 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.314092    2566 scope.go:117] "RemoveContainer" containerID="b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276"
May 25 11:42:53.314384 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:42:53.314362    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276\": container with ID starting with b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276 not found: ID does not exist" containerID="b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276"
May 25 11:42:53.314461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.314395    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276"} err="failed to get container status \"b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276\": rpc error: code = NotFound desc = could not find container \"b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276\": container with ID starting with b08f8c48e35a4fbc476d6688cc7969db234015666f4f0eb38d9ae3a68780b276 not found: ID does not exist"
May 25 11:42:53.314461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.314418    2566 scope.go:117] "RemoveContainer" containerID="00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d"
May 25 11:42:53.314994 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:42:53.314962    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d\": container with ID starting with 00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d not found: ID does not exist" containerID="00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d"
May 25 11:42:53.315054 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.314996    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d"} err="failed to get container status \"00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d\": rpc error: code = NotFound desc = could not find container \"00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d\": container with ID starting with 00920d6eb3ec312346df31e7ce294a85343c4b87f607d4d61a8e620a73c6bc2d not found: ID does not exist"
May 25 11:42:53.316268 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.316249    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-68f4449pqg"]
May 25 11:42:53.836586 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:42:53.836532    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643b3354-7e20-4b44-af49-174ff81a728a" path="/var/lib/kubelet/pods/643b3354-7e20-4b44-af49-174ff81a728a/volumes"
May 25 11:43:01.773313 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773270    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"]
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773745    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="main"
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773764    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="main"
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773786    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="tokenizer"
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773792    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="tokenizer"
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773807    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="storage-initializer"
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773814    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="storage-initializer"
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773871    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="main"
May 25 11:43:01.775913 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.773879    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="643b3354-7e20-4b44-af49-174ff81a728a" containerName="tokenizer"
May 25 11:43:01.776910 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.776890    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.779184 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.779165    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-5xkwb\""
May 25 11:43:01.779291 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.779165    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 25 11:43:01.788965 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.788941    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"]
May 25 11:43:01.847421 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.847387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.847653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.847448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.847653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.847481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957cz\" (UniqueName: \"kubernetes.io/projected/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kube-api-access-957cz\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.847653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.847515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.847653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.847621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.847834 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.847671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948161 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948122    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948359 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948184    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-957cz\" (UniqueName: \"kubernetes.io/projected/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kube-api-access-957cz\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948428 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948353    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948487 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948442    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948575 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948498    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948575 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948531    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948575 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948563    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948887 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948827    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.948887 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948872    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.949089 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.948890    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.951047 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.951028    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:01.957481 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:01.957457    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-957cz\" (UniqueName: \"kubernetes.io/projected/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kube-api-access-957cz\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:02.086034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:02.085948    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:02.231401 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:02.231367    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"]
May 25 11:43:02.232847 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:43:02.232816    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6c0ecf_f389_43bb_b6b6_7206d3cf9f77.slice/crio-9da3ec68a973da0473932f2c3d55be7624216cc725e7a9ffe93d27a3fef5c833 WatchSource:0}: Error finding container 9da3ec68a973da0473932f2c3d55be7624216cc725e7a9ffe93d27a3fef5c833: Status 404 returned error can't find the container with id 9da3ec68a973da0473932f2c3d55be7624216cc725e7a9ffe93d27a3fef5c833
May 25 11:43:02.328347 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:02.328296    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerStarted","Data":"ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33"}
May 25 11:43:02.328347 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:02.328352    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerStarted","Data":"9da3ec68a973da0473932f2c3d55be7624216cc725e7a9ffe93d27a3fef5c833"}
May 25 11:43:03.333456 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:03.333361    2566 generic.go:358] "Generic (PLEG): container finished" podID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerID="ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33" exitCode=0
May 25 11:43:03.333456 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:03.333411    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerDied","Data":"ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33"}
May 25 11:43:04.339600 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:04.339562    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerStarted","Data":"23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90"}
May 25 11:43:04.339600 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:04.339602    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerStarted","Data":"1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2"}
May 25 11:43:04.340132 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:04.339665    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:04.362090 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:04.362035    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" podStartSLOduration=3.36201924 podStartE2EDuration="3.36201924s" podCreationTimestamp="2026-05-25 11:43:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:43:04.360395114 +0000 UTC m=+1543.115371911" watchObservedRunningTime="2026-05-25 11:43:04.36201924 +0000 UTC m=+1543.116996078"
May 25 11:43:12.086221 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:12.086188    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:12.086221 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:12.086224    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:12.088885 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:12.088865    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:12.371456 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:12.371426    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:43:33.375796 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:43:33.375767    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:44:16.472679 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:16.472641    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"]
May 25 11:44:16.473241 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:16.473014    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="main" containerID="cri-o://a524d0c5e155e428f105aa208d4f479a03027b06b0b4369bd42314c8337e24eb" gracePeriod=30
May 25 11:44:16.473241 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:16.473045    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="tokenizer" containerID="cri-o://f4b9373d5a210ecd9742c970ef33fef5b5867fd579cd203774da4ce86abf9a93" gracePeriod=30
May 25 11:44:16.607586 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:16.607520    2566 generic.go:358] "Generic (PLEG): container finished" podID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerID="a524d0c5e155e428f105aa208d4f479a03027b06b0b4369bd42314c8337e24eb" exitCode=0
May 25 11:44:16.607749 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:16.607582    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerDied","Data":"a524d0c5e155e428f105aa208d4f479a03027b06b0b4369bd42314c8337e24eb"}
May 25 11:44:17.613823 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.613785    2566 generic.go:358] "Generic (PLEG): container finished" podID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerID="f4b9373d5a210ecd9742c970ef33fef5b5867fd579cd203774da4ce86abf9a93" exitCode=0
May 25 11:44:17.614192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.613854    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerDied","Data":"f4b9373d5a210ecd9742c970ef33fef5b5867fd579cd203774da4ce86abf9a93"}
May 25 11:44:17.731693 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.731666    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:44:17.785195 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785166    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-cache\") pod \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") "
May 25 11:44:17.785353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785210    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt496\" (UniqueName: \"kubernetes.io/projected/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kube-api-access-dt496\") pod \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") "
May 25 11:44:17.785353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785242    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-tmp\") pod \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") "
May 25 11:44:17.785353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785259    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-uds\") pod \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") "
May 25 11:44:17.785353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785288    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tls-certs\") pod \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") "
May 25 11:44:17.785353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785308    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kserve-provision-location\") pod \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\" (UID: \"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41\") "
May 25 11:44:17.785707 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785496    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" (UID: "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:17.785707 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785558    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" (UID: "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:17.785707 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.785614    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" (UID: "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:17.786214 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.786190    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" (UID: "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:17.787453 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.787431    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" (UID: "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:44:17.787529 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.787463    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kube-api-access-dt496" (OuterVolumeSpecName: "kube-api-access-dt496") pod "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" (UID: "a8662dbf-dfc4-43c2-93fe-dd0a53c54b41"). InnerVolumeSpecName "kube-api-access-dt496". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:44:17.886460 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.886428    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:44:17.886460 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.886458    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt496\" (UniqueName: \"kubernetes.io/projected/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kube-api-access-dt496\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:44:17.886460 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.886469    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:44:17.886708 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.886478    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:44:17.886708 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.886487    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:44:17.886708 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:17.886495    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:44:18.619929 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:18.619893    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb" event={"ID":"a8662dbf-dfc4-43c2-93fe-dd0a53c54b41","Type":"ContainerDied","Data":"e478df580afd6e15b8950e6814e90aa8f58bfbcf22a51fa615d266ced995a05c"}
May 25 11:44:18.619929 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:18.619921    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"
May 25 11:44:18.620466 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:18.619946    2566 scope.go:117] "RemoveContainer" containerID="f4b9373d5a210ecd9742c970ef33fef5b5867fd579cd203774da4ce86abf9a93"
May 25 11:44:18.629192 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:18.629172    2566 scope.go:117] "RemoveContainer" containerID="a524d0c5e155e428f105aa208d4f479a03027b06b0b4369bd42314c8337e24eb"
May 25 11:44:18.637909 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:18.637882    2566 scope.go:117] "RemoveContainer" containerID="542b81e4b8877f188f0a894177040dbc81dd1187836c0271fc4601bf63971408"
May 25 11:44:18.638510 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:18.638486    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"]
May 25 11:44:18.642874 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:18.642850    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779d968kvb"]
May 25 11:44:19.835689 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:19.835658    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" path="/var/lib/kubelet/pods/a8662dbf-dfc4-43c2-93fe-dd0a53c54b41/volumes"
May 25 11:44:28.445409 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445375    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"]
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445739    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="tokenizer"
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445751    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="tokenizer"
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445760    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="main"
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445766    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="main"
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445781    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="storage-initializer"
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445787    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="storage-initializer"
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445836    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="tokenizer"
May 25 11:44:28.445932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.445847    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="a8662dbf-dfc4-43c2-93fe-dd0a53c54b41" containerName="main"
May 25 11:44:28.450698 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.450677    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.453184 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.453154    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 25 11:44:28.453309 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.453154    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-xqmtq\""
May 25 11:44:28.463289 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.463256    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"]
May 25 11:44:28.581153 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.581115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.581345 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.581172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.581345 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.581265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2r77\" (UniqueName: \"kubernetes.io/projected/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kube-api-access-d2r77\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.581345 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.581302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.581518 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.581389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.581518 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.581423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.683056 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.683017    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d2r77\" (UniqueName: \"kubernetes.io/projected/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kube-api-access-d2r77\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.683351 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.683328    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.683525 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.683510    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.683674 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.683659    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.683813 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.683799    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.683938 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.683919    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.684499 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.684473    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.685261 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.685238    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.688562 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.688482    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.688737 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.688713    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.688822 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.688733    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.693086 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.693060    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2r77\" (UniqueName: \"kubernetes.io/projected/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kube-api-access-d2r77\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.760284 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.760199    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:28.908501 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:28.908471    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"]
May 25 11:44:28.910238 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:44:28.910209    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd729c5d_bdbf_4bda_b855_9fbbaf8c74b5.slice/crio-7eb73fa71518ad17267e071d2d62b166a744d9d893573952b02a5af892bbc5b2 WatchSource:0}: Error finding container 7eb73fa71518ad17267e071d2d62b166a744d9d893573952b02a5af892bbc5b2: Status 404 returned error can't find the container with id 7eb73fa71518ad17267e071d2d62b166a744d9d893573952b02a5af892bbc5b2
May 25 11:44:29.662083 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:29.662043    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerStarted","Data":"fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec"}
May 25 11:44:29.662083 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:29.662082    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerStarted","Data":"7eb73fa71518ad17267e071d2d62b166a744d9d893573952b02a5af892bbc5b2"}
May 25 11:44:30.667272 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:30.667240    2566 generic.go:358] "Generic (PLEG): container finished" podID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerID="fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec" exitCode=0
May 25 11:44:30.667778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:30.667318    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerDied","Data":"fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec"}
May 25 11:44:31.673744 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:31.673707    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerStarted","Data":"69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3"}
May 25 11:44:31.673744 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:31.673749    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerStarted","Data":"52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199"}
May 25 11:44:31.674152 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:31.673837    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:31.704440 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:31.704393    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" podStartSLOduration=3.704379131 podStartE2EDuration="3.704379131s" podCreationTimestamp="2026-05-25 11:44:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:44:31.701835369 +0000 UTC m=+1630.456812168" watchObservedRunningTime="2026-05-25 11:44:31.704379131 +0000 UTC m=+1630.459355926"
May 25 11:44:38.761146 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:38.761105    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:38.761146 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:38.761145    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:38.764098 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:38.764073    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:44:39.705981 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:44:39.705949    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:45:00.709274 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:00.709244    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:45:01.885845 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:01.885811    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"]
May 25 11:45:01.886299 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:01.886127    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="main" containerID="cri-o://52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199" gracePeriod=30
May 25 11:45:01.886299 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:01.886171    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="tokenizer" containerID="cri-o://69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3" gracePeriod=30
May 25 11:45:02.791112 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:02.791074    2566 generic.go:358] "Generic (PLEG): container finished" podID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerID="52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199" exitCode=0
May 25 11:45:02.791296 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:02.791131    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerDied","Data":"52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199"}
May 25 11:45:03.241443 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.241416    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:45:03.265329 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265296    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-cache\") pod \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") "
May 25 11:45:03.265430 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265359    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-tmp\") pod \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") "
May 25 11:45:03.265430 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265384    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tls-certs\") pod \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") "
May 25 11:45:03.265430 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265405    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-uds\") pod \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") "
May 25 11:45:03.265430 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265424    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kserve-provision-location\") pod \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") "
May 25 11:45:03.265661 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265447    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2r77\" (UniqueName: \"kubernetes.io/projected/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kube-api-access-d2r77\") pod \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\" (UID: \"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5\") "
May 25 11:45:03.265661 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265615    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" (UID: "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:03.265785 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265686    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" (UID: "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:03.265843 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.265796    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" (UID: "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:03.266354 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.266307    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" (UID: "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:03.267641 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.267615    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" (UID: "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:45:03.267848 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.267827    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kube-api-access-d2r77" (OuterVolumeSpecName: "kube-api-access-d2r77") pod "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" (UID: "dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5"). InnerVolumeSpecName "kube-api-access-d2r77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:45:03.366686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.366595    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:03.366686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.366626    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:03.366686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.366635    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:03.366686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.366645    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2r77\" (UniqueName: \"kubernetes.io/projected/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-kube-api-access-d2r77\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:03.366686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.366654    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:03.366686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.366663    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:03.796230 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.796139    2566 generic.go:358] "Generic (PLEG): container finished" podID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerID="69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3" exitCode=0
May 25 11:45:03.796230 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.796205    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerDied","Data":"69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3"}
May 25 11:45:03.796456 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.796229    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"
May 25 11:45:03.796456 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.796237    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q" event={"ID":"dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5","Type":"ContainerDied","Data":"7eb73fa71518ad17267e071d2d62b166a744d9d893573952b02a5af892bbc5b2"}
May 25 11:45:03.796456 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.796254    2566 scope.go:117] "RemoveContainer" containerID="69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3"
May 25 11:45:03.805312 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.805284    2566 scope.go:117] "RemoveContainer" containerID="52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199"
May 25 11:45:03.813151 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.813130    2566 scope.go:117] "RemoveContainer" containerID="fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec"
May 25 11:45:03.819461 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.819432    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"]
May 25 11:45:03.821847 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.821821    2566 scope.go:117] "RemoveContainer" containerID="69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3"
May 25 11:45:03.822159 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:45:03.822129    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3\": container with ID starting with 69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3 not found: ID does not exist" containerID="69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3"
May 25 11:45:03.822261 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.822168    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3"} err="failed to get container status \"69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3\": rpc error: code = NotFound desc = could not find container \"69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3\": container with ID starting with 69bda9de26e37bdcc42dd6f9c4f41ff461ec033a8e98ff7e8e915753610205f3 not found: ID does not exist"
May 25 11:45:03.822261 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.822196    2566 scope.go:117] "RemoveContainer" containerID="52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199"
May 25 11:45:03.822516 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:45:03.822472    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199\": container with ID starting with 52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199 not found: ID does not exist" containerID="52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199"
May 25 11:45:03.822592 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.822526    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199"} err="failed to get container status \"52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199\": rpc error: code = NotFound desc = could not find container \"52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199\": container with ID starting with 52f03698202a64cd4dc7d051d7d86d9b6e755ec6b4e299de33d9c87113fbc199 not found: ID does not exist"
May 25 11:45:03.822592 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.822569    2566 scope.go:117] "RemoveContainer" containerID="fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec"
May 25 11:45:03.822848 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:45:03.822828    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec\": container with ID starting with fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec not found: ID does not exist" containerID="fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec"
May 25 11:45:03.822916 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.822858    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec"} err="failed to get container status \"fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec\": rpc error: code = NotFound desc = could not find container \"fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec\": container with ID starting with fe89aaffd2368b231b77a68deb287a32cb3349a0a46fcad4baa6e100a9bb23ec not found: ID does not exist"
May 25 11:45:03.823757 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.823737    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-57755zvl5q"]
May 25 11:45:03.835651 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:03.835629    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" path="/var/lib/kubelet/pods/dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5/volumes"
May 25 11:45:06.588898 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.588858    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"]
May 25 11:45:06.589272 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589230    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="main"
May 25 11:45:06.589272 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589244    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="main"
May 25 11:45:06.589272 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589263    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="tokenizer"
May 25 11:45:06.589272 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589268    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="tokenizer"
May 25 11:45:06.589402 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589279    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="storage-initializer"
May 25 11:45:06.589402 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589285    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="storage-initializer"
May 25 11:45:06.589402 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589343    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="main"
May 25 11:45:06.589402 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.589352    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="dd729c5d-bdbf-4bda-b855-9fbbaf8c74b5" containerName="tokenizer"
May 25 11:45:06.594327 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.594300    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.597014 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.596991    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 25 11:45:06.597202 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.597188    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-d4dmc\""
May 25 11:45:06.610926 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.610891    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"]
May 25 11:45:06.691932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.691886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlx8\" (UniqueName: \"kubernetes.io/projected/94877947-4ce0-4946-afab-23a041549636-kube-api-access-mzlx8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.691932 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.691936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.692161 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.691962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.692161 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.692041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.692161 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.692073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.692161 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.692095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94877947-4ce0-4946-afab-23a041549636-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793161 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793118    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793355 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793170    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793355 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793205    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94877947-4ce0-4946-afab-23a041549636-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793355 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793306    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlx8\" (UniqueName: \"kubernetes.io/projected/94877947-4ce0-4946-afab-23a041549636-kube-api-access-mzlx8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793534 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793364    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793534 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793412    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793668 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793616    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.793871 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793847    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.794040 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.793926    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.794195 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.794174    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.796377 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.796355    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94877947-4ce0-4946-afab-23a041549636-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.805990 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.805968    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlx8\" (UniqueName: \"kubernetes.io/projected/94877947-4ce0-4946-afab-23a041549636-kube-api-access-mzlx8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:06.905123 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:06.905021    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:07.058702 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:07.058671    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"]
May 25 11:45:07.060127 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:45:07.060092    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94877947_4ce0_4946_afab_23a041549636.slice/crio-88118c7155358dce4a016d620f59c460fc31bdd60a175164dcea692cc96aec6e WatchSource:0}: Error finding container 88118c7155358dce4a016d620f59c460fc31bdd60a175164dcea692cc96aec6e: Status 404 returned error can't find the container with id 88118c7155358dce4a016d620f59c460fc31bdd60a175164dcea692cc96aec6e
May 25 11:45:07.813507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:07.813475    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerStarted","Data":"63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555"}
May 25 11:45:07.813507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:07.813511    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerStarted","Data":"88118c7155358dce4a016d620f59c460fc31bdd60a175164dcea692cc96aec6e"}
May 25 11:45:08.818820 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:08.818784    2566 generic.go:358] "Generic (PLEG): container finished" podID="94877947-4ce0-4946-afab-23a041549636" containerID="63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555" exitCode=0
May 25 11:45:08.819219 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:08.818872    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerDied","Data":"63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555"}
May 25 11:45:09.826181 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:09.826137    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerStarted","Data":"32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be"}
May 25 11:45:09.826181 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:09.826184    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerStarted","Data":"a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c"}
May 25 11:45:09.826730 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:09.826286    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:09.850977 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:09.850921    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" podStartSLOduration=3.850906002 podStartE2EDuration="3.850906002s" podCreationTimestamp="2026-05-25 11:45:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:45:09.847938926 +0000 UTC m=+1668.602915759" watchObservedRunningTime="2026-05-25 11:45:09.850906002 +0000 UTC m=+1668.605882846"
May 25 11:45:16.905628 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:16.905587    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:16.905628 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:16.905636    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:16.908335 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:16.908311    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:17.856342 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:17.856302    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:38.860986 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:38.860916    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:40.371902 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:40.371871    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"]
May 25 11:45:40.372994 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:40.372932    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="main" containerID="cri-o://a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c" gracePeriod=30
May 25 11:45:40.373430 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:40.373377    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="tokenizer" containerID="cri-o://32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be" gracePeriod=30
May 25 11:45:40.941944 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:40.941912    2566 generic.go:358] "Generic (PLEG): container finished" podID="94877947-4ce0-4946-afab-23a041549636" containerID="a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c" exitCode=0
May 25 11:45:40.942311 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:40.941975    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerDied","Data":"a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c"}
May 25 11:45:41.724921 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.724897    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:41.807278 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807190    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-uds\") pod \"94877947-4ce0-4946-afab-23a041549636\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") "
May 25 11:45:41.807278 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807244    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzlx8\" (UniqueName: \"kubernetes.io/projected/94877947-4ce0-4946-afab-23a041549636-kube-api-access-mzlx8\") pod \"94877947-4ce0-4946-afab-23a041549636\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") "
May 25 11:45:41.807509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807298    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-kserve-provision-location\") pod \"94877947-4ce0-4946-afab-23a041549636\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") "
May 25 11:45:41.807509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807363    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94877947-4ce0-4946-afab-23a041549636-tls-certs\") pod \"94877947-4ce0-4946-afab-23a041549636\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") "
May 25 11:45:41.807509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807401    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-cache\") pod \"94877947-4ce0-4946-afab-23a041549636\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") "
May 25 11:45:41.807509 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807449    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-tmp\") pod \"94877947-4ce0-4946-afab-23a041549636\" (UID: \"94877947-4ce0-4946-afab-23a041549636\") "
May 25 11:45:41.807738 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807516    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "94877947-4ce0-4946-afab-23a041549636" (UID: "94877947-4ce0-4946-afab-23a041549636"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:41.807808 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807769    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "94877947-4ce0-4946-afab-23a041549636" (UID: "94877947-4ce0-4946-afab-23a041549636"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:41.807933 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.807901    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "94877947-4ce0-4946-afab-23a041549636" (UID: "94877947-4ce0-4946-afab-23a041549636"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:41.808217 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.808190    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94877947-4ce0-4946-afab-23a041549636" (UID: "94877947-4ce0-4946-afab-23a041549636"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:45:41.809575 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.809526    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94877947-4ce0-4946-afab-23a041549636-kube-api-access-mzlx8" (OuterVolumeSpecName: "kube-api-access-mzlx8") pod "94877947-4ce0-4946-afab-23a041549636" (UID: "94877947-4ce0-4946-afab-23a041549636"). InnerVolumeSpecName "kube-api-access-mzlx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:45:41.809575 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.809559    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94877947-4ce0-4946-afab-23a041549636-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94877947-4ce0-4946-afab-23a041549636" (UID: "94877947-4ce0-4946-afab-23a041549636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:45:41.908464 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.908432    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:41.908464 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.908461    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:41.908710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.908473    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:41.908710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.908487    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlx8\" (UniqueName: \"kubernetes.io/projected/94877947-4ce0-4946-afab-23a041549636-kube-api-access-mzlx8\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:41.908710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.908500    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94877947-4ce0-4946-afab-23a041549636-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:41.908710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.908515    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94877947-4ce0-4946-afab-23a041549636-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:45:41.947011 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.946977    2566 generic.go:358] "Generic (PLEG): container finished" podID="94877947-4ce0-4946-afab-23a041549636" containerID="32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be" exitCode=0
May 25 11:45:41.947174 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.947050    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"
May 25 11:45:41.947174 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.947066    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerDied","Data":"32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be"}
May 25 11:45:41.947174 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.947099    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s" event={"ID":"94877947-4ce0-4946-afab-23a041549636","Type":"ContainerDied","Data":"88118c7155358dce4a016d620f59c460fc31bdd60a175164dcea692cc96aec6e"}
May 25 11:45:41.947174 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.947114    2566 scope.go:117] "RemoveContainer" containerID="32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be"
May 25 11:45:41.958279 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.958236    2566 scope.go:117] "RemoveContainer" containerID="a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c"
May 25 11:45:41.965361 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.965336    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"]
May 25 11:45:41.967560 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.967527    2566 scope.go:117] "RemoveContainer" containerID="63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555"
May 25 11:45:41.970087 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.970062    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fd8546btw4s"]
May 25 11:45:41.975444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.975425    2566 scope.go:117] "RemoveContainer" containerID="32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be"
May 25 11:45:41.975728 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:45:41.975706    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be\": container with ID starting with 32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be not found: ID does not exist" containerID="32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be"
May 25 11:45:41.975813 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.975735    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be"} err="failed to get container status \"32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be\": rpc error: code = NotFound desc = could not find container \"32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be\": container with ID starting with 32d4cac196ca7e04c44ffef475657de4a3747681145df5fbd20161912a16a8be not found: ID does not exist"
May 25 11:45:41.975813 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.975754    2566 scope.go:117] "RemoveContainer" containerID="a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c"
May 25 11:45:41.976015 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:45:41.975998    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c\": container with ID starting with a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c not found: ID does not exist" containerID="a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c"
May 25 11:45:41.976063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.976022    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c"} err="failed to get container status \"a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c\": rpc error: code = NotFound desc = could not find container \"a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c\": container with ID starting with a2fec6b6a17d9615e4b82e84e215af0fd4f2a8bb713052df06d9d27ee9410e6c not found: ID does not exist"
May 25 11:45:41.976063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.976041    2566 scope.go:117] "RemoveContainer" containerID="63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555"
May 25 11:45:41.976254 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:45:41.976229    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555\": container with ID starting with 63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555 not found: ID does not exist" containerID="63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555"
May 25 11:45:41.976300 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:41.976250    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555"} err="failed to get container status \"63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555\": rpc error: code = NotFound desc = could not find container \"63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555\": container with ID starting with 63eba81b4da024b39184a9a5d04cd4b0c5616277ba9570ebc10aa742bf0e3555 not found: ID does not exist"
May 25 11:45:43.838688 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:43.838651    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94877947-4ce0-4946-afab-23a041549636" path="/var/lib/kubelet/pods/94877947-4ce0-4946-afab-23a041549636/volumes"
May 25 11:45:47.770384 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770345    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"]
May 25 11:45:47.770818 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770736    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="main"
May 25 11:45:47.770818 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770754    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="main"
May 25 11:45:47.770818 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770772    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="tokenizer"
May 25 11:45:47.770818 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770781    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="tokenizer"
May 25 11:45:47.770818 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770794    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="storage-initializer"
May 25 11:45:47.770818 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770803    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="storage-initializer"
May 25 11:45:47.771198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770873    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="main"
May 25 11:45:47.771198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.770883    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="94877947-4ce0-4946-afab-23a041549636" containerName="tokenizer"
May 25 11:45:47.774490 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.774465    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.777160 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.777135    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 25 11:45:47.790361 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.790334    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"]
May 25 11:45:47.852789 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.852754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tmp-dir\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.852964 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.852800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-model-cache\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.852964 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.852873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.852964 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.852924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gms7\" (UniqueName: \"kubernetes.io/projected/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kube-api-access-6gms7\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.852964 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.852956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-dshm\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.853126 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.852974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-home\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.853126 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.852993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.953840 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.953805    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-model-cache\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954048 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.953866    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954048 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.953925    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6gms7\" (UniqueName: \"kubernetes.io/projected/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kube-api-access-6gms7\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954048 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.953963    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-dshm\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954048 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.953989    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-home\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954048 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.954013    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954048 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.954044    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tmp-dir\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954372 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.954262    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-model-cache\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954372 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.954345    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954372 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.954362    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tmp-dir\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.954534 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.954489    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-home\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.956588 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.956564    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.956725 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.956681    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-dshm\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:47.963034 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:47.963005    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gms7\" (UniqueName: \"kubernetes.io/projected/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kube-api-access-6gms7\") pod \"precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:48.088610 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.087221    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:45:48.091176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.091146    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"]
May 25 11:45:48.097397 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.097370    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.100133 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.099944    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-4r9q2\""
May 25 11:45:48.121787 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.121751    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"]
May 25 11:45:48.155822 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.155789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzc7r\" (UniqueName: \"kubernetes.io/projected/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kube-api-access-zzc7r\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.155965 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.155826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.155965 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.155873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.155965 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.155896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.155965 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.155916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.155965 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.155949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.256886 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.256768    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.256886 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.256811    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.256886 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.256836    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.256886 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.256882    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.257139 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.256948    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zzc7r\" (UniqueName: \"kubernetes.io/projected/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kube-api-access-zzc7r\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.257139 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.256975    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.257250 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.257208    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.257306 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.257252    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.257465 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.257420    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"]
May 25 11:45:48.257638 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.257614    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.257760 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.257641    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.259560 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.259524    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.260285 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:45:48.260256    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94869429_37b8_41ae_bc45_9e73f5b0e6a9.slice/crio-92631eb98492fab3da5c3ee5221a6f1163817a61e425e8f715e39086060ab3eb WatchSource:0}: Error finding container 92631eb98492fab3da5c3ee5221a6f1163817a61e425e8f715e39086060ab3eb: Status 404 returned error can't find the container with id 92631eb98492fab3da5c3ee5221a6f1163817a61e425e8f715e39086060ab3eb
May 25 11:45:48.262391 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.262378    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:45:48.272971 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.272946    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzc7r\" (UniqueName: \"kubernetes.io/projected/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kube-api-access-zzc7r\") pod \"precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.417061 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.416959    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:48.556407 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.556363    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"]
May 25 11:45:48.557412 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:45:48.557386    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f903e6a_47ac_4245_889d_7d9efb5bc4ca.slice/crio-fd056059e9738810f20d53f4930be1c84e09da14798deaff3607ed60279f7204 WatchSource:0}: Error finding container fd056059e9738810f20d53f4930be1c84e09da14798deaff3607ed60279f7204: Status 404 returned error can't find the container with id fd056059e9738810f20d53f4930be1c84e09da14798deaff3607ed60279f7204
May 25 11:45:48.972830 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.972791    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerStarted","Data":"8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb"}
May 25 11:45:48.972830 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.972832    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerStarted","Data":"fd056059e9738810f20d53f4930be1c84e09da14798deaff3607ed60279f7204"}
May 25 11:45:48.974296 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.974263    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" event={"ID":"94869429-37b8-41ae-bc45-9e73f5b0e6a9","Type":"ContainerStarted","Data":"964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067"}
May 25 11:45:48.974407 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:48.974305    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" event={"ID":"94869429-37b8-41ae-bc45-9e73f5b0e6a9","Type":"ContainerStarted","Data":"92631eb98492fab3da5c3ee5221a6f1163817a61e425e8f715e39086060ab3eb"}
May 25 11:45:49.980390 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:49.980343    2566 generic.go:358] "Generic (PLEG): container finished" podID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerID="8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb" exitCode=0
May 25 11:45:49.980849 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:49.980434    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerDied","Data":"8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb"}
May 25 11:45:50.987385 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:50.987349    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerStarted","Data":"0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f"}
May 25 11:45:50.987845 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:50.987400    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerStarted","Data":"fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe"}
May 25 11:45:51.017200 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:51.017145    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" podStartSLOduration=3.017129233 podStartE2EDuration="3.017129233s" podCreationTimestamp="2026-05-25 11:45:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:45:51.014223125 +0000 UTC m=+1709.769199923" watchObservedRunningTime="2026-05-25 11:45:51.017129233 +0000 UTC m=+1709.772106028"
May 25 11:45:51.992180 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:51.992143    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:58.418062 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:58.418024    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:58.418062 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:58.418069    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:58.419209 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:45:58.419177    2566 logging.go:55] [core] [Channel #1013 SubChannel #1014]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.48:9003", ServerName: "10.132.0.48:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.48:9003: connect: connection refused"
May 25 11:45:58.420423 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:58.420406    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:59.028712 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:59.028682    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:45:59.418594 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:45:59.418528    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.48:9003\" within 1s: context deadline exceeded"
May 25 11:46:08.418060 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:46:08.418024    2566 logging.go:55] [core] [Channel #1021 SubChannel #1022]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.48:9003", ServerName: "10.132.0.48:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.48:9003: connect: connection refused"
May 25 11:46:09.418682 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:09.418634    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.48:9003\" within 1s: context deadline exceeded"
May 25 11:46:20.033488 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:20.033458    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:46:43.207310 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:43.207275    2566 generic.go:358] "Generic (PLEG): container finished" podID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerID="964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067" exitCode=0
May 25 11:46:43.207773 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:43.207336    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" event={"ID":"94869429-37b8-41ae-bc45-9e73f5b0e6a9","Type":"ContainerDied","Data":"964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067"}
May 25 11:46:45.216374 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:45.216335    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" event={"ID":"94869429-37b8-41ae-bc45-9e73f5b0e6a9","Type":"ContainerStarted","Data":"54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057"}
May 25 11:46:45.238222 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:45.238170    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" podStartSLOduration=57.205801079 podStartE2EDuration="58.238152045s" podCreationTimestamp="2026-05-25 11:45:47 +0000 UTC" firstStartedPulling="2026-05-25 11:46:43.208499985 +0000 UTC m=+1761.963476773" lastFinishedPulling="2026-05-25 11:46:44.240850964 +0000 UTC m=+1762.995827739" observedRunningTime="2026-05-25 11:46:45.236166389 +0000 UTC m=+1763.991143184" watchObservedRunningTime="2026-05-25 11:46:45.238152045 +0000 UTC m=+1763.993128841"
May 25 11:46:48.088371 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:48.088325    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:46:48.088835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:48.088470    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:46:48.101125 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:48.101094    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:46:48.239115 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:48.239086    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:46:49.532893 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:49.532854    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"]
May 25 11:46:49.548699 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:49.548666    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"]
May 25 11:46:49.549054 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:49.549028    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="main" containerID="cri-o://fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe" gracePeriod=30
May 25 11:46:49.549184 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:49.549141    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="tokenizer" containerID="cri-o://0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f" gracePeriod=30
May 25 11:46:50.032772 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:46:50.032736    2566 logging.go:55] [core] [Channel #1068 SubChannel #1069]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.48:9003", ServerName: "10.132.0.48:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.48:9003: connect: connection refused"
May 25 11:46:50.236655 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:50.236614    2566 generic.go:358] "Generic (PLEG): container finished" podID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerID="fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe" exitCode=0
May 25 11:46:50.236838 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:50.236687    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerDied","Data":"fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe"}
May 25 11:46:50.311768 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:50.311665    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs: secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 25 11:46:50.311768 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:50.311736    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs podName:94869429-37b8-41ae-bc45-9e73f5b0e6a9 nodeName:}" failed. No retries permitted until 2026-05-25 11:46:50.811719943 +0000 UTC m=+1769.566696717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs") pod "precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9") : secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 25 11:46:50.815714 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:50.815680    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs: secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 25 11:46:50.816112 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:50.815768    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs podName:94869429-37b8-41ae-bc45-9e73f5b0e6a9 nodeName:}" failed. No retries permitted until 2026-05-25 11:46:51.815749457 +0000 UTC m=+1770.570726230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs") pod "precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9") : secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 25 11:46:51.032756 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.032705    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.48:9003\" within 1s: context deadline exceeded"
May 25 11:46:51.200088 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.200060    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:46:51.242029 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.241991    2566 generic.go:358] "Generic (PLEG): container finished" podID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerID="0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f" exitCode=0
May 25 11:46:51.242185 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.242080    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerDied","Data":"0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f"}
May 25 11:46:51.242185 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.242101    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"
May 25 11:46:51.242185 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.242126    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf" event={"ID":"2f903e6a-47ac-4245-889d-7d9efb5bc4ca","Type":"ContainerDied","Data":"fd056059e9738810f20d53f4930be1c84e09da14798deaff3607ed60279f7204"}
May 25 11:46:51.242185 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.242149    2566 scope.go:117] "RemoveContainer" containerID="0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f"
May 25 11:46:51.242571 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.242470    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" podUID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerName="main" containerID="cri-o://54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057" gracePeriod=30
May 25 11:46:51.252659 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.252632    2566 scope.go:117] "RemoveContainer" containerID="fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe"
May 25 11:46:51.319193 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319089    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzc7r\" (UniqueName: \"kubernetes.io/projected/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kube-api-access-zzc7r\") pod \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") "
May 25 11:46:51.319193 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319140    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-cache\") pod \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") "
May 25 11:46:51.319425 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319218    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kserve-provision-location\") pod \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") "
May 25 11:46:51.319425 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319249    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tls-certs\") pod \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") "
May 25 11:46:51.319425 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319273    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-uds\") pod \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") "
May 25 11:46:51.319425 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319300    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-tmp\") pod \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\" (UID: \"2f903e6a-47ac-4245-889d-7d9efb5bc4ca\") "
May 25 11:46:51.319692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319565    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2f903e6a-47ac-4245-889d-7d9efb5bc4ca" (UID: "2f903e6a-47ac-4245-889d-7d9efb5bc4ca"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.319692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319594    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "2f903e6a-47ac-4245-889d-7d9efb5bc4ca" (UID: "2f903e6a-47ac-4245-889d-7d9efb5bc4ca"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.319803 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.319775    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "2f903e6a-47ac-4245-889d-7d9efb5bc4ca" (UID: "2f903e6a-47ac-4245-889d-7d9efb5bc4ca"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.320097 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.320075    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f903e6a-47ac-4245-889d-7d9efb5bc4ca" (UID: "2f903e6a-47ac-4245-889d-7d9efb5bc4ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.321377 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.321353    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f903e6a-47ac-4245-889d-7d9efb5bc4ca" (UID: "2f903e6a-47ac-4245-889d-7d9efb5bc4ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:46:51.321377 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.321362    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kube-api-access-zzc7r" (OuterVolumeSpecName: "kube-api-access-zzc7r") pod "2f903e6a-47ac-4245-889d-7d9efb5bc4ca" (UID: "2f903e6a-47ac-4245-889d-7d9efb5bc4ca"). InnerVolumeSpecName "kube-api-access-zzc7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:46:51.353792 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.353764    2566 scope.go:117] "RemoveContainer" containerID="8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb"
May 25 11:46:51.362313 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.362290    2566 scope.go:117] "RemoveContainer" containerID="0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f"
May 25 11:46:51.362688 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:51.362667    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f\": container with ID starting with 0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f not found: ID does not exist" containerID="0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f"
May 25 11:46:51.362739 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.362701    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f"} err="failed to get container status \"0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f\": rpc error: code = NotFound desc = could not find container \"0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f\": container with ID starting with 0243b055f08b0b43e6a46f89358ff2e3d5685edf905128f236cc425971bf8e1f not found: ID does not exist"
May 25 11:46:51.362739 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.362724    2566 scope.go:117] "RemoveContainer" containerID="fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe"
May 25 11:46:51.363000 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:51.362974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe\": container with ID starting with fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe not found: ID does not exist" containerID="fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe"
May 25 11:46:51.363104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.363006    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe"} err="failed to get container status \"fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe\": rpc error: code = NotFound desc = could not find container \"fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe\": container with ID starting with fdcf75c83af8f07b247f27bbeb881254d92abc2f0949923fceaaaba89e592dbe not found: ID does not exist"
May 25 11:46:51.363104 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.363025    2566 scope.go:117] "RemoveContainer" containerID="8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb"
May 25 11:46:51.363297 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:51.363279    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb\": container with ID starting with 8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb not found: ID does not exist" containerID="8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb"
May 25 11:46:51.363340 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.363303    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb"} err="failed to get container status \"8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb\": rpc error: code = NotFound desc = could not find container \"8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb\": container with ID starting with 8a90c55c269f6be62be737942a9de4c229b03b345af36f8d3166c317193369bb not found: ID does not exist"
May 25 11:46:51.420221 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.420191    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.420221 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.420221    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzc7r\" (UniqueName: \"kubernetes.io/projected/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kube-api-access-zzc7r\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.420436 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.420232    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.420436 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.420243    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.420436 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.420257    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.420436 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.420272    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2f903e6a-47ac-4245-889d-7d9efb5bc4ca-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.501626 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.501603    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:46:51.565974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.565941    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"]
May 25 11:46:51.571451 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.571389    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-55c84b88cgjjf"]
May 25 11:46:51.621531 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621500    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kserve-provision-location\") pod \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") "
May 25 11:46:51.621531 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621534    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-home\") pod \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") "
May 25 11:46:51.621774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621657    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tmp-dir\") pod \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") "
May 25 11:46:51.621774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621687    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-dshm\") pod \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") "
May 25 11:46:51.621774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621732    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-model-cache\") pod \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") "
May 25 11:46:51.621774 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621760    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gms7\" (UniqueName: \"kubernetes.io/projected/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kube-api-access-6gms7\") pod \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") "
May 25 11:46:51.621986 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621808    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs\") pod \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\" (UID: \"94869429-37b8-41ae-bc45-9e73f5b0e6a9\") "
May 25 11:46:51.621986 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621869    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-home" (OuterVolumeSpecName: "home") pod "94869429-37b8-41ae-bc45-9e73f5b0e6a9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.622088 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.621979    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "94869429-37b8-41ae-bc45-9e73f5b0e6a9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.622161 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.622138    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.622225 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.622153    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94869429-37b8-41ae-bc45-9e73f5b0e6a9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.622225 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.622163    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.624058 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.624029    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kube-api-access-6gms7" (OuterVolumeSpecName: "kube-api-access-6gms7") pod "94869429-37b8-41ae-bc45-9e73f5b0e6a9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9"). InnerVolumeSpecName "kube-api-access-6gms7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:46:51.624058 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.624053    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-dshm" (OuterVolumeSpecName: "dshm") pod "94869429-37b8-41ae-bc45-9e73f5b0e6a9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.624258 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.624244    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94869429-37b8-41ae-bc45-9e73f5b0e6a9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:46:51.681292 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.681248    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94869429-37b8-41ae-bc45-9e73f5b0e6a9" (UID: "94869429-37b8-41ae-bc45-9e73f5b0e6a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:51.723176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.723135    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.723176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.723170    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.723176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.723181    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gms7\" (UniqueName: \"kubernetes.io/projected/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kube-api-access-6gms7\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.723406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.723190    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94869429-37b8-41ae-bc45-9e73f5b0e6a9-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.723406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.723200    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94869429-37b8-41ae-bc45-9e73f5b0e6a9-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:46:51.836211 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:51.836130    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" path="/var/lib/kubelet/pods/2f903e6a-47ac-4245-889d-7d9efb5bc4ca/volumes"
May 25 11:46:52.247249 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.247214    2566 generic.go:358] "Generic (PLEG): container finished" podID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerID="54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057" exitCode=0
May 25 11:46:52.247444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.247288    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"
May 25 11:46:52.247444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.247285    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" event={"ID":"94869429-37b8-41ae-bc45-9e73f5b0e6a9","Type":"ContainerDied","Data":"54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057"}
May 25 11:46:52.247444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.247400    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9" event={"ID":"94869429-37b8-41ae-bc45-9e73f5b0e6a9","Type":"ContainerDied","Data":"92631eb98492fab3da5c3ee5221a6f1163817a61e425e8f715e39086060ab3eb"}
May 25 11:46:52.247444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.247441    2566 scope.go:117] "RemoveContainer" containerID="54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057"
May 25 11:46:52.256574 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.256530    2566 scope.go:117] "RemoveContainer" containerID="964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067"
May 25 11:46:52.267601 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.267574    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"]
May 25 11:46:52.273105 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.273076    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-58dc66bdf5-vz4h9"]
May 25 11:46:52.323193 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.323164    2566 scope.go:117] "RemoveContainer" containerID="54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057"
May 25 11:46:52.323523 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:52.323495    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057\": container with ID starting with 54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057 not found: ID does not exist" containerID="54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057"
May 25 11:46:52.323728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.323533    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057"} err="failed to get container status \"54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057\": rpc error: code = NotFound desc = could not find container \"54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057\": container with ID starting with 54854f47e2dd46696a4804b2613131103ba58fb9f78f76d8a2c72e63bb842057 not found: ID does not exist"
May 25 11:46:52.323728 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.323575    2566 scope.go:117] "RemoveContainer" containerID="964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067"
May 25 11:46:52.323864 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:46:52.323842    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067\": container with ID starting with 964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067 not found: ID does not exist" containerID="964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067"
May 25 11:46:52.323934 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:52.323869    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067"} err="failed to get container status \"964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067\": rpc error: code = NotFound desc = could not find container \"964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067\": container with ID starting with 964521ece01fec76e9c282774d059934a2b771a1dafdc1ba1fd68249e00f6067 not found: ID does not exist"
May 25 11:46:53.835678 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:46:53.835644    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" path="/var/lib/kubelet/pods/94869429-37b8-41ae-bc45-9e73f5b0e6a9/volumes"
May 25 11:47:40.625289 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:40.625243    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"]
May 25 11:47:40.625781 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:40.625613    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="main" containerID="cri-o://1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2" gracePeriod=30
May 25 11:47:40.625863 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:40.625739    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="tokenizer" containerID="cri-o://23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90" gracePeriod=30
May 25 11:47:41.436063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.436025    2566 generic.go:358] "Generic (PLEG): container finished" podID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerID="1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2" exitCode=0
May 25 11:47:41.436236 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.436084    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerDied","Data":"1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2"}
May 25 11:47:41.883448 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.883424    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:47:41.975196 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975101    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kserve-provision-location\") pod \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") "
May 25 11:47:41.975196 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975184    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-uds\") pod \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") "
May 25 11:47:41.975407 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975208    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-cache\") pod \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") "
May 25 11:47:41.975407 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975258    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-tmp\") pod \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") "
May 25 11:47:41.975407 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975278    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-957cz\" (UniqueName: \"kubernetes.io/projected/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kube-api-access-957cz\") pod \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") "
May 25 11:47:41.975407 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975304    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tls-certs\") pod \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\" (UID: \"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77\") "
May 25 11:47:41.975658 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975466    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" (UID: "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:41.975658 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975509    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" (UID: "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:41.975658 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975610    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" (UID: "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:41.975899 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.975877    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" (UID: "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:47:41.977404 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.977378    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kube-api-access-957cz" (OuterVolumeSpecName: "kube-api-access-957cz") pod "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" (UID: "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77"). InnerVolumeSpecName "kube-api-access-957cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:47:41.977518 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:41.977401    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" (UID: "4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:47:42.076551 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.076518    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:47:42.076721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.076565    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:47:42.076721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.076576    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957cz\" (UniqueName: \"kubernetes.io/projected/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kube-api-access-957cz\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:47:42.076721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.076587    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:47:42.076721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.076595    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:47:42.076721 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.076604    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:47:42.441278 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.441243    2566 generic.go:358] "Generic (PLEG): container finished" podID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerID="23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90" exitCode=0
May 25 11:47:42.441467 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.441310    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"
May 25 11:47:42.441467 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.441325    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerDied","Data":"23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90"}
May 25 11:47:42.441467 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.441359    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7" event={"ID":"4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77","Type":"ContainerDied","Data":"9da3ec68a973da0473932f2c3d55be7624216cc725e7a9ffe93d27a3fef5c833"}
May 25 11:47:42.441467 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.441374    2566 scope.go:117] "RemoveContainer" containerID="23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90"
May 25 11:47:42.449870 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.449852    2566 scope.go:117] "RemoveContainer" containerID="1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2"
May 25 11:47:42.457581 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.457564    2566 scope.go:117] "RemoveContainer" containerID="ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33"
May 25 11:47:42.465957 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.465927    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"]
May 25 11:47:42.466057 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.465934    2566 scope.go:117] "RemoveContainer" containerID="23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90"
May 25 11:47:42.466281 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:47:42.466258    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90\": container with ID starting with 23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90 not found: ID does not exist" containerID="23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90"
May 25 11:47:42.466354 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.466292    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90"} err="failed to get container status \"23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90\": rpc error: code = NotFound desc = could not find container \"23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90\": container with ID starting with 23e941e29ee29305da33a9396b0540b50bb0b0b7f773f53f7342d4bf6ba7df90 not found: ID does not exist"
May 25 11:47:42.466354 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.466312    2566 scope.go:117] "RemoveContainer" containerID="1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2"
May 25 11:47:42.466581 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:47:42.466562    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2\": container with ID starting with 1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2 not found: ID does not exist" containerID="1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2"
May 25 11:47:42.466657 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.466590    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2"} err="failed to get container status \"1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2\": rpc error: code = NotFound desc = could not find container \"1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2\": container with ID starting with 1fa179159841aa50bfdc5175f1968677347619708a4d1731b75a975168e306f2 not found: ID does not exist"
May 25 11:47:42.466657 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.466611    2566 scope.go:117] "RemoveContainer" containerID="ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33"
May 25 11:47:42.466879 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:47:42.466854    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33\": container with ID starting with ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33 not found: ID does not exist" containerID="ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33"
May 25 11:47:42.466920 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.466875    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33"} err="failed to get container status \"ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33\": rpc error: code = NotFound desc = could not find container \"ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33\": container with ID starting with ef35f7b63cc20faf5f3979a282353d04c1ebc486fdd1cda54e464a186b2a2e33 not found: ID does not exist"
May 25 11:47:42.470958 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:42.470937    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7967c6dffgv5p7"]
May 25 11:47:43.835835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:47:43.835802    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" path="/var/lib/kubelet/pods/4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77/volumes"
May 25 11:49:45.589305 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589261    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589756    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="tokenizer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589775    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="tokenizer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589786    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="tokenizer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589795    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="tokenizer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589807    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589815    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589828    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="storage-initializer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589837    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="storage-initializer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589848    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="storage-initializer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589857    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="storage-initializer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589878    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589886    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589900    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589907    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589921    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerName="storage-initializer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.589930    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerName="storage-initializer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.590020    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="tokenizer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.590032    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b6c0ecf-f389-43bb-b6b6-7206d3cf9f77" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.590041    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="main"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.590052    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f903e6a-47ac-4245-889d-7d9efb5bc4ca" containerName="tokenizer"
May 25 11:49:45.590063 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.590061    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="94869429-37b8-41ae-bc45-9e73f5b0e6a9" containerName="main"
May 25 11:49:45.593531 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.593510    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.596071 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.596054    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-bnwh4\""
May 25 11:49:45.596893 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.596875    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5cxfn\""
May 25 11:49:45.597024 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.597007    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 25 11:49:45.603934 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.603907    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 11:49:45.668508 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.668478    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"]
May 25 11:49:45.671920 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.671899    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.674172 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674153    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-7hlc6\""
May 25 11:49:45.674295 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.674295 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.674295 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.674444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.674503 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.674503 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84hw\" (UniqueName: \"kubernetes.io/projected/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kube-api-access-n84hw\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.674636 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.674532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.688837 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.688813    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"]
May 25 11:49:45.775781 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.775744    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.775781 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.775780    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n84hw\" (UniqueName: \"kubernetes.io/projected/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kube-api-access-n84hw\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.775998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.775812    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.775998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.775848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.775998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.775890    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.775998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.775952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.775998 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.775993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvwn\" (UniqueName: \"kubernetes.io/projected/147a4cb2-83bf-4753-a6fb-756ed530167a-kube-api-access-pvvwn\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776067    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776099    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776112    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776145    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776168    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147a4cb2-83bf-4753-a6fb-756ed530167a-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.776232 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.776624 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776270    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.776624 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.776460    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.778206 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.778181    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.778333 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.778311    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.785894 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.785867    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84hw\" (UniqueName: \"kubernetes.io/projected/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kube-api-access-n84hw\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.877288 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877199    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877288 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877240    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877288 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877263    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvwn\" (UniqueName: \"kubernetes.io/projected/147a4cb2-83bf-4753-a6fb-756ed530167a-kube-api-access-pvvwn\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877619 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877301    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147a4cb2-83bf-4753-a6fb-756ed530167a-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877619 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877323    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877619 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877360    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877757 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877701    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877757 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877701    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877748    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.877835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.877778    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.879769 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.879751    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147a4cb2-83bf-4753-a6fb-756ed530167a-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.886350 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.886323    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvwn\" (UniqueName: \"kubernetes.io/projected/147a4cb2-83bf-4753-a6fb-756ed530167a-kube-api-access-pvvwn\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:45.904042 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.904019    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:49:45.981841 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:45.981796    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:46.045342 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:46.045312    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 11:49:46.046787 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:49:46.046756    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc7f0c2_c024_4d76_8f44_69a09b00eec2.slice/crio-6724511697b7b45d8d683295647e5b2bf30660a201298d377512f6c5566ef3b7 WatchSource:0}: Error finding container 6724511697b7b45d8d683295647e5b2bf30660a201298d377512f6c5566ef3b7: Status 404 returned error can't find the container with id 6724511697b7b45d8d683295647e5b2bf30660a201298d377512f6c5566ef3b7
May 25 11:49:46.124222 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:46.124191    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"]
May 25 11:49:46.125966 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:49:46.125924    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147a4cb2_83bf_4753_a6fb_756ed530167a.slice/crio-3baf0a3ed57baeeabec6b9d800a085170d6197e5cae7df7365a90851f1703b5f WatchSource:0}: Error finding container 3baf0a3ed57baeeabec6b9d800a085170d6197e5cae7df7365a90851f1703b5f: Status 404 returned error can't find the container with id 3baf0a3ed57baeeabec6b9d800a085170d6197e5cae7df7365a90851f1703b5f
May 25 11:49:46.890501 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:46.890459    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"0bc7f0c2-c024-4d76-8f44-69a09b00eec2","Type":"ContainerStarted","Data":"4b8a1d15f1bc1a3843dff3a9eeab9d1cb6896435045f236b53333cb2c657aa20"}
May 25 11:49:46.890501 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:46.890503    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"0bc7f0c2-c024-4d76-8f44-69a09b00eec2","Type":"ContainerStarted","Data":"6724511697b7b45d8d683295647e5b2bf30660a201298d377512f6c5566ef3b7"}
May 25 11:49:46.891775 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:46.891753    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerStarted","Data":"7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533"}
May 25 11:49:46.891894 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:46.891779    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerStarted","Data":"3baf0a3ed57baeeabec6b9d800a085170d6197e5cae7df7365a90851f1703b5f"}
May 25 11:49:47.897040 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:47.896999    2566 generic.go:358] "Generic (PLEG): container finished" podID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerID="7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533" exitCode=0
May 25 11:49:47.897521 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:47.897077    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerDied","Data":"7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533"}
May 25 11:49:48.906093 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:48.906057    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerStarted","Data":"5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9"}
May 25 11:49:48.906460 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:48.906100    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerStarted","Data":"7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c"}
May 25 11:49:48.906460 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:48.906129    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:48.929221 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:48.929162    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" podStartSLOduration=3.929149062 podStartE2EDuration="3.929149062s" podCreationTimestamp="2026-05-25 11:49:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:49:48.9272819 +0000 UTC m=+1947.682258697" watchObservedRunningTime="2026-05-25 11:49:48.929149062 +0000 UTC m=+1947.684125879"
May 25 11:49:51.919674 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:51.919644    2566 generic.go:358] "Generic (PLEG): container finished" podID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerID="4b8a1d15f1bc1a3843dff3a9eeab9d1cb6896435045f236b53333cb2c657aa20" exitCode=0
May 25 11:49:51.919987 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:51.919713    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"0bc7f0c2-c024-4d76-8f44-69a09b00eec2","Type":"ContainerDied","Data":"4b8a1d15f1bc1a3843dff3a9eeab9d1cb6896435045f236b53333cb2c657aa20"}
May 25 11:49:55.982328 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:55.982145    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:55.982328 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:55.982197    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:49:55.983663 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:49:55.983628    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="tokenizer" probeResult="failure" output="Get \"http://10.132.0.50:8082/healthz\": dial tcp 10.132.0.50:8082: connect: connection refused"
May 25 11:50:05.984176 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:50:05.984066    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:50:05.985893 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:50:05.985828    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:50:26.989656 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:50:26.989625    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:50:31.085091 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:50:31.085012    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"0bc7f0c2-c024-4d76-8f44-69a09b00eec2","Type":"ContainerStarted","Data":"59841150e8663aa51e5143e4e66be8ea99b256da23829e4476443325e809745a"}
May 25 11:50:31.103663 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:50:31.103512    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podStartSLOduration=7.224606137 podStartE2EDuration="46.103494549s" podCreationTimestamp="2026-05-25 11:49:45 +0000 UTC" firstStartedPulling="2026-05-25 11:49:51.920929823 +0000 UTC m=+1950.675906597" lastFinishedPulling="2026-05-25 11:50:30.799818232 +0000 UTC m=+1989.554795009" observedRunningTime="2026-05-25 11:50:31.102794395 +0000 UTC m=+1989.857771214" watchObservedRunningTime="2026-05-25 11:50:31.103494549 +0000 UTC m=+1989.858471346"
May 25 11:51:43.910535 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:43.910443    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"]
May 25 11:51:43.911119 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:43.910868    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="main" containerID="cri-o://7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c" gracePeriod=30
May 25 11:51:43.911119 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:43.910932    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="tokenizer" containerID="cri-o://5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9" gracePeriod=30
May 25 11:51:44.372972 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:44.372936    2566 generic.go:358] "Generic (PLEG): container finished" podID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerID="7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c" exitCode=0
May 25 11:51:44.373163 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:44.373014    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerDied","Data":"7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c"}
May 25 11:51:45.160202 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.160177    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:51:45.231008 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.230923    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-tmp\") pod \"147a4cb2-83bf-4753-a6fb-756ed530167a\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") "
May 25 11:51:45.231008 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.230994    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-kserve-provision-location\") pod \"147a4cb2-83bf-4753-a6fb-756ed530167a\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") "
May 25 11:51:45.231228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231041    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-uds\") pod \"147a4cb2-83bf-4753-a6fb-756ed530167a\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") "
May 25 11:51:45.231228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231069    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvvwn\" (UniqueName: \"kubernetes.io/projected/147a4cb2-83bf-4753-a6fb-756ed530167a-kube-api-access-pvvwn\") pod \"147a4cb2-83bf-4753-a6fb-756ed530167a\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") "
May 25 11:51:45.231228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231115    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-cache\") pod \"147a4cb2-83bf-4753-a6fb-756ed530167a\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") "
May 25 11:51:45.231228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231153    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147a4cb2-83bf-4753-a6fb-756ed530167a-tls-certs\") pod \"147a4cb2-83bf-4753-a6fb-756ed530167a\" (UID: \"147a4cb2-83bf-4753-a6fb-756ed530167a\") "
May 25 11:51:45.231444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231327    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "147a4cb2-83bf-4753-a6fb-756ed530167a" (UID: "147a4cb2-83bf-4753-a6fb-756ed530167a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:45.231444 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231349    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "147a4cb2-83bf-4753-a6fb-756ed530167a" (UID: "147a4cb2-83bf-4753-a6fb-756ed530167a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:45.231572 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231447    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-uds\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:51:45.231572 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231467    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-tmp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:51:45.231812 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.231758    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "147a4cb2-83bf-4753-a6fb-756ed530167a" (UID: "147a4cb2-83bf-4753-a6fb-756ed530167a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:45.232112 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.232085    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "147a4cb2-83bf-4753-a6fb-756ed530167a" (UID: "147a4cb2-83bf-4753-a6fb-756ed530167a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:51:45.233422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.233394    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147a4cb2-83bf-4753-a6fb-756ed530167a-kube-api-access-pvvwn" (OuterVolumeSpecName: "kube-api-access-pvvwn") pod "147a4cb2-83bf-4753-a6fb-756ed530167a" (UID: "147a4cb2-83bf-4753-a6fb-756ed530167a"). InnerVolumeSpecName "kube-api-access-pvvwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:51:45.233422 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.233399    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147a4cb2-83bf-4753-a6fb-756ed530167a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "147a4cb2-83bf-4753-a6fb-756ed530167a" (UID: "147a4cb2-83bf-4753-a6fb-756ed530167a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:51:45.318008 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.317971    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 11:51:45.318307 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.318284    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podUID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerName="main" containerID="cri-o://59841150e8663aa51e5143e4e66be8ea99b256da23829e4476443325e809745a" gracePeriod=30
May 25 11:51:45.332606 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.332576    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:51:45.332724 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.332614    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvwn\" (UniqueName: \"kubernetes.io/projected/147a4cb2-83bf-4753-a6fb-756ed530167a-kube-api-access-pvvwn\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:51:45.332724 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.332632    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/147a4cb2-83bf-4753-a6fb-756ed530167a-tokenizer-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:51:45.332724 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.332646    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147a4cb2-83bf-4753-a6fb-756ed530167a-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:51:45.379680 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.379650    2566 generic.go:358] "Generic (PLEG): container finished" podID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerID="5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9" exitCode=0
May 25 11:51:45.379804 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.379730    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerDied","Data":"5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9"}
May 25 11:51:45.379804 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.379748    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"
May 25 11:51:45.379804 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.379770    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh" event={"ID":"147a4cb2-83bf-4753-a6fb-756ed530167a","Type":"ContainerDied","Data":"3baf0a3ed57baeeabec6b9d800a085170d6197e5cae7df7365a90851f1703b5f"}
May 25 11:51:45.379804 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.379791    2566 scope.go:117] "RemoveContainer" containerID="5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9"
May 25 11:51:45.388240 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.388220    2566 scope.go:117] "RemoveContainer" containerID="7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c"
May 25 11:51:45.397978 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.397962    2566 scope.go:117] "RemoveContainer" containerID="7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533"
May 25 11:51:45.405293 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.405271    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"]
May 25 11:51:45.406526 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.406508    2566 scope.go:117] "RemoveContainer" containerID="5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9"
May 25 11:51:45.406886 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:51:45.406836    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9\": container with ID starting with 5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9 not found: ID does not exist" containerID="5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9"
May 25 11:51:45.406989 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.406876    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9"} err="failed to get container status \"5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9\": rpc error: code = NotFound desc = could not find container \"5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9\": container with ID starting with 5f6a776112ce47e89e819fed3d778c247ea2c9d85b04b894933938e8281fa9a9 not found: ID does not exist"
May 25 11:51:45.406989 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.406901    2566 scope.go:117] "RemoveContainer" containerID="7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c"
May 25 11:51:45.407406 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:51:45.407382    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c\": container with ID starting with 7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c not found: ID does not exist" containerID="7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c"
May 25 11:51:45.407507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.407418    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c"} err="failed to get container status \"7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c\": rpc error: code = NotFound desc = could not find container \"7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c\": container with ID starting with 7cd8fd6597d62d9508864d6f0b62beabed0c1f6a4243fe79f9188d5997945b8c not found: ID does not exist"
May 25 11:51:45.407507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.407440    2566 scope.go:117] "RemoveContainer" containerID="7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533"
May 25 11:51:45.407754 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:51:45.407726    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533\": container with ID starting with 7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533 not found: ID does not exist" containerID="7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533"
May 25 11:51:45.407844 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.407760    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533"} err="failed to get container status \"7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533\": rpc error: code = NotFound desc = could not find container \"7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533\": container with ID starting with 7a6a481e2f1813cd57c98368f103f0d8af90058a6672ca6e992fb746d5f17533 not found: ID does not exist"
May 25 11:51:45.409449 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.409432    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche9w4jh"]
May 25 11:51:45.836160 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:51:45.836116    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" path="/var/lib/kubelet/pods/147a4cb2-83bf-4753-a6fb-756ed530167a/volumes"
May 25 11:52:15.493557 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.493512    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_0bc7f0c2-c024-4d76-8f44-69a09b00eec2/main/0.log"
May 25 11:52:15.493981 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.493899    2566 generic.go:358] "Generic (PLEG): container finished" podID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerID="59841150e8663aa51e5143e4e66be8ea99b256da23829e4476443325e809745a" exitCode=137
May 25 11:52:15.494040 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.493974    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"0bc7f0c2-c024-4d76-8f44-69a09b00eec2","Type":"ContainerDied","Data":"59841150e8663aa51e5143e4e66be8ea99b256da23829e4476443325e809745a"}
May 25 11:52:15.976603 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.976580    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_0bc7f0c2-c024-4d76-8f44-69a09b00eec2/main/0.log"
May 25 11:52:15.976974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.976958    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:52:15.990016 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.989992    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tls-certs\") pod \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") "
May 25 11:52:15.990123 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990024    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tmp-dir\") pod \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") "
May 25 11:52:15.990123 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990050    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n84hw\" (UniqueName: \"kubernetes.io/projected/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kube-api-access-n84hw\") pod \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") "
May 25 11:52:15.990123 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990084    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-model-cache\") pod \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") "
May 25 11:52:15.990238 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990133    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-home\") pod \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") "
May 25 11:52:15.990238 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990180    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kserve-provision-location\") pod \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") "
May 25 11:52:15.990238 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990226    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-dshm\") pod \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\" (UID: \"0bc7f0c2-c024-4d76-8f44-69a09b00eec2\") "
May 25 11:52:15.990429 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990397    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bc7f0c2-c024-4d76-8f44-69a09b00eec2" (UID: "0bc7f0c2-c024-4d76-8f44-69a09b00eec2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:52:15.990517 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990491    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:52:15.990916 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.990887    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-home" (OuterVolumeSpecName: "home") pod "0bc7f0c2-c024-4d76-8f44-69a09b00eec2" (UID: "0bc7f0c2-c024-4d76-8f44-69a09b00eec2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:52:15.992351 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.992324    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bc7f0c2-c024-4d76-8f44-69a09b00eec2" (UID: "0bc7f0c2-c024-4d76-8f44-69a09b00eec2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:52:15.992592 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.992564    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kube-api-access-n84hw" (OuterVolumeSpecName: "kube-api-access-n84hw") pod "0bc7f0c2-c024-4d76-8f44-69a09b00eec2" (UID: "0bc7f0c2-c024-4d76-8f44-69a09b00eec2"). InnerVolumeSpecName "kube-api-access-n84hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:52:15.992754 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:15.992733    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-dshm" (OuterVolumeSpecName: "dshm") pod "0bc7f0c2-c024-4d76-8f44-69a09b00eec2" (UID: "0bc7f0c2-c024-4d76-8f44-69a09b00eec2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:52:16.005969 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.005924    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bc7f0c2-c024-4d76-8f44-69a09b00eec2" (UID: "0bc7f0c2-c024-4d76-8f44-69a09b00eec2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:52:16.054455 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.054407    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bc7f0c2-c024-4d76-8f44-69a09b00eec2" (UID: "0bc7f0c2-c024-4d76-8f44-69a09b00eec2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:52:16.091507 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.091478    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kserve-provision-location\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:52:16.091659 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.091509    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:52:16.091659 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.091522    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:52:16.091659 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.091531    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:52:16.091659 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.091566    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84hw\" (UniqueName: \"kubernetes.io/projected/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-kube-api-access-n84hw\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:52:16.091659 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.091578    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc7f0c2-c024-4d76-8f44-69a09b00eec2-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:52:16.499976 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.499889    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_0bc7f0c2-c024-4d76-8f44-69a09b00eec2/main/0.log"
May 25 11:52:16.500363 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.500235    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"0bc7f0c2-c024-4d76-8f44-69a09b00eec2","Type":"ContainerDied","Data":"6724511697b7b45d8d683295647e5b2bf30660a201298d377512f6c5566ef3b7"}
May 25 11:52:16.500363 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.500275    2566 scope.go:117] "RemoveContainer" containerID="59841150e8663aa51e5143e4e66be8ea99b256da23829e4476443325e809745a"
May 25 11:52:16.500363 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.500310    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 25 11:52:16.509111 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.509089    2566 scope.go:117] "RemoveContainer" containerID="4b8a1d15f1bc1a3843dff3a9eeab9d1cb6896435045f236b53333cb2c657aa20"
May 25 11:52:16.523964 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.523937    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 11:52:16.526975 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:16.526949    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 25 11:52:17.835341 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:52:17.835310    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" path="/var/lib/kubelet/pods/0bc7f0c2-c024-4d76-8f44-69a09b00eec2/volumes"
May 25 11:53:11.870454 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870420    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"]
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870828    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="main"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870847    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="main"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870861    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="tokenizer"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870869    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="tokenizer"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870898    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="storage-initializer"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870907    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="storage-initializer"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870919    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerName="storage-initializer"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870927    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerName="storage-initializer"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870939    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerName="main"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.870986    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerName="main"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.871056    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="0bc7f0c2-c024-4d76-8f44-69a09b00eec2" containerName="main"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.871066    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="main"
May 25 11:53:11.872940 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.871072    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="147a4cb2-83bf-4753-a6fb-756ed530167a" containerName="tokenizer"
May 25 11:53:11.873726 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.873711    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:11.881352 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.881328    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"]
May 25 11:53:11.939198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.939159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9tbf\" (UniqueName: \"kubernetes.io/projected/159887ab-b2ff-4ec4-9571-f363baef9f4d-kube-api-access-v9tbf\") pod \"llmisvc-controller-manager-84bbbfb696-p4wtr\" (UID: \"159887ab-b2ff-4ec4-9571-f363baef9f4d\") " pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:11.939198 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:11.939204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/159887ab-b2ff-4ec4-9571-f363baef9f4d-cert\") pod \"llmisvc-controller-manager-84bbbfb696-p4wtr\" (UID: \"159887ab-b2ff-4ec4-9571-f363baef9f4d\") " pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:12.039994 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.039940    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9tbf\" (UniqueName: \"kubernetes.io/projected/159887ab-b2ff-4ec4-9571-f363baef9f4d-kube-api-access-v9tbf\") pod \"llmisvc-controller-manager-84bbbfb696-p4wtr\" (UID: \"159887ab-b2ff-4ec4-9571-f363baef9f4d\") " pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:12.039994 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.040004    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/159887ab-b2ff-4ec4-9571-f363baef9f4d-cert\") pod \"llmisvc-controller-manager-84bbbfb696-p4wtr\" (UID: \"159887ab-b2ff-4ec4-9571-f363baef9f4d\") " pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:12.042225 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.042206    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/159887ab-b2ff-4ec4-9571-f363baef9f4d-cert\") pod \"llmisvc-controller-manager-84bbbfb696-p4wtr\" (UID: \"159887ab-b2ff-4ec4-9571-f363baef9f4d\") " pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:12.047966 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.047940    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9tbf\" (UniqueName: \"kubernetes.io/projected/159887ab-b2ff-4ec4-9571-f363baef9f4d-kube-api-access-v9tbf\") pod \"llmisvc-controller-manager-84bbbfb696-p4wtr\" (UID: \"159887ab-b2ff-4ec4-9571-f363baef9f4d\") " pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:12.184388 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.184296    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:12.310532 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.310508    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"]
May 25 11:53:12.312029 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:53:12.312001    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod159887ab_b2ff_4ec4_9571_f363baef9f4d.slice/crio-06972a7d125a106bc7881aea067344d177da0d53f60f04fec8b00234a43bae26 WatchSource:0}: Error finding container 06972a7d125a106bc7881aea067344d177da0d53f60f04fec8b00234a43bae26: Status 404 returned error can't find the container with id 06972a7d125a106bc7881aea067344d177da0d53f60f04fec8b00234a43bae26
May 25 11:53:12.313275 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.313258    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:53:12.707826 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:12.707786    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr" event={"ID":"159887ab-b2ff-4ec4-9571-f363baef9f4d","Type":"ContainerStarted","Data":"06972a7d125a106bc7881aea067344d177da0d53f60f04fec8b00234a43bae26"}
May 25 11:53:13.712713 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:13.712676    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr" event={"ID":"159887ab-b2ff-4ec4-9571-f363baef9f4d","Type":"ContainerStarted","Data":"c36cd8f7f52a9d781fd9bbdc15ec60640133b676dc1b4c14f2e2710ffb8c7852"}
May 25 11:53:13.713113 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:13.712734    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:13.729189 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:13.729142    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr" podStartSLOduration=2.241483177 podStartE2EDuration="2.729127565s" podCreationTimestamp="2026-05-25 11:53:11 +0000 UTC" firstStartedPulling="2026-05-25 11:53:12.313386504 +0000 UTC m=+2151.068363278" lastFinishedPulling="2026-05-25 11:53:12.801030889 +0000 UTC m=+2151.556007666" observedRunningTime="2026-05-25 11:53:13.727678992 +0000 UTC m=+2152.482655788" watchObservedRunningTime="2026-05-25 11:53:13.729127565 +0000 UTC m=+2152.484104362"
May 25 11:53:44.717852 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:44.717822    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84bbbfb696-p4wtr"
May 25 11:53:44.761314 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:44.761281    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"]
May 25 11:53:44.761578 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:44.761516    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg" podUID="f144da74-d67f-4b88-9ea1-37d100500c91" containerName="manager" containerID="cri-o://87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be" gracePeriod=30
May 25 11:53:44.997149 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:44.997125    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:53:45.093736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.093698    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xgv8\" (UniqueName: \"kubernetes.io/projected/f144da74-d67f-4b88-9ea1-37d100500c91-kube-api-access-6xgv8\") pod \"f144da74-d67f-4b88-9ea1-37d100500c91\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") "
May 25 11:53:45.093736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.093739    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f144da74-d67f-4b88-9ea1-37d100500c91-cert\") pod \"f144da74-d67f-4b88-9ea1-37d100500c91\" (UID: \"f144da74-d67f-4b88-9ea1-37d100500c91\") "
May 25 11:53:45.095738 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.095707    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f144da74-d67f-4b88-9ea1-37d100500c91-cert" (OuterVolumeSpecName: "cert") pod "f144da74-d67f-4b88-9ea1-37d100500c91" (UID: "f144da74-d67f-4b88-9ea1-37d100500c91"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:53:45.095888 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.095846    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f144da74-d67f-4b88-9ea1-37d100500c91-kube-api-access-6xgv8" (OuterVolumeSpecName: "kube-api-access-6xgv8") pod "f144da74-d67f-4b88-9ea1-37d100500c91" (UID: "f144da74-d67f-4b88-9ea1-37d100500c91"). InnerVolumeSpecName "kube-api-access-6xgv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:53:45.194642 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.194604    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xgv8\" (UniqueName: \"kubernetes.io/projected/f144da74-d67f-4b88-9ea1-37d100500c91-kube-api-access-6xgv8\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:53:45.194642 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.194637    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f144da74-d67f-4b88-9ea1-37d100500c91-cert\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:53:45.834591 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.834556    2566 generic.go:358] "Generic (PLEG): container finished" podID="f144da74-d67f-4b88-9ea1-37d100500c91" containerID="87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be" exitCode=0
May 25 11:53:45.835003 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.834669    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"
May 25 11:53:45.835103 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.835083    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg" event={"ID":"f144da74-d67f-4b88-9ea1-37d100500c91","Type":"ContainerDied","Data":"87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be"}
May 25 11:53:45.835155 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.835111    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-c557f5ddb-xwbrg" event={"ID":"f144da74-d67f-4b88-9ea1-37d100500c91","Type":"ContainerDied","Data":"a5fc621f63b200433ec2b523ec048ebb8da39822126b189543d53e3fae179f3a"}
May 25 11:53:45.835155 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.835126    2566 scope.go:117] "RemoveContainer" containerID="87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be"
May 25 11:53:45.845190 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.845172    2566 scope.go:117] "RemoveContainer" containerID="87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be"
May 25 11:53:45.845517 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:53:45.845484    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be\": container with ID starting with 87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be not found: ID does not exist" containerID="87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be"
May 25 11:53:45.845649 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.845515    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be"} err="failed to get container status \"87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be\": rpc error: code = NotFound desc = could not find container \"87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be\": container with ID starting with 87383be0aeaf13f78f481de764d480cb02f255327d683be1def9b961a2a6d1be not found: ID does not exist"
May 25 11:53:45.859263 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.859235    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"]
May 25 11:53:45.864297 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:45.864276    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-c557f5ddb-xwbrg"]
May 25 11:53:47.835523 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:53:47.835490    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f144da74-d67f-4b88-9ea1-37d100500c91" path="/var/lib/kubelet/pods/f144da74-d67f-4b88-9ea1-37d100500c91/volumes"
May 25 11:54:10.870068 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.870032    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l4z78/must-gather-cfmj8"]
May 25 11:54:10.870535 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.870355    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f144da74-d67f-4b88-9ea1-37d100500c91" containerName="manager"
May 25 11:54:10.870535 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.870366    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f144da74-d67f-4b88-9ea1-37d100500c91" containerName="manager"
May 25 11:54:10.870535 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.870426    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="f144da74-d67f-4b88-9ea1-37d100500c91" containerName="manager"
May 25 11:54:10.873382 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.873365    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:10.876022 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.875995    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l4z78\"/\"openshift-service-ca.crt\""
May 25 11:54:10.876146 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.876063    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l4z78\"/\"default-dockercfg-pbcb5\""
May 25 11:54:10.876955 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.876937    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l4z78\"/\"kube-root-ca.crt\""
May 25 11:54:10.883716 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:10.883692    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l4z78/must-gather-cfmj8"]
May 25 11:54:11.006798 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.006764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh889\" (UniqueName: \"kubernetes.io/projected/e0165b7c-fb5d-4956-adfa-348a67f93f8b-kube-api-access-fh889\") pod \"must-gather-cfmj8\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") " pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:11.006964 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.006808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0165b7c-fb5d-4956-adfa-348a67f93f8b-must-gather-output\") pod \"must-gather-cfmj8\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") " pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:11.107321 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.107284    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fh889\" (UniqueName: \"kubernetes.io/projected/e0165b7c-fb5d-4956-adfa-348a67f93f8b-kube-api-access-fh889\") pod \"must-gather-cfmj8\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") " pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:11.107484 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.107330    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0165b7c-fb5d-4956-adfa-348a67f93f8b-must-gather-output\") pod \"must-gather-cfmj8\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") " pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:11.107640 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.107624    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0165b7c-fb5d-4956-adfa-348a67f93f8b-must-gather-output\") pod \"must-gather-cfmj8\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") " pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:11.115643 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.115620    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh889\" (UniqueName: \"kubernetes.io/projected/e0165b7c-fb5d-4956-adfa-348a67f93f8b-kube-api-access-fh889\") pod \"must-gather-cfmj8\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") " pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:11.182923 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.182844    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:11.305047 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.305021    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l4z78/must-gather-cfmj8"]
May 25 11:54:11.306770 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:54:11.306743    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0165b7c_fb5d_4956_adfa_348a67f93f8b.slice/crio-afa1df19829cf2893c03c02dbc2a83394624cbb550bd2d893ad6609bb6796f68 WatchSource:0}: Error finding container afa1df19829cf2893c03c02dbc2a83394624cbb550bd2d893ad6609bb6796f68: Status 404 returned error can't find the container with id afa1df19829cf2893c03c02dbc2a83394624cbb550bd2d893ad6609bb6796f68
May 25 11:54:11.928870 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:11.928834    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l4z78/must-gather-cfmj8" event={"ID":"e0165b7c-fb5d-4956-adfa-348a67f93f8b","Type":"ContainerStarted","Data":"afa1df19829cf2893c03c02dbc2a83394624cbb550bd2d893ad6609bb6796f68"}
May 25 11:54:17.958915 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:17.958872    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l4z78/must-gather-cfmj8" event={"ID":"e0165b7c-fb5d-4956-adfa-348a67f93f8b","Type":"ContainerStarted","Data":"91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239"}
May 25 11:54:17.959394 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:17.958922    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l4z78/must-gather-cfmj8" event={"ID":"e0165b7c-fb5d-4956-adfa-348a67f93f8b","Type":"ContainerStarted","Data":"b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9"}
May 25 11:54:17.978084 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:17.978025    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-l4z78/must-gather-cfmj8" podStartSLOduration=1.931111912 podStartE2EDuration="7.978007571s" podCreationTimestamp="2026-05-25 11:54:10 +0000 UTC" firstStartedPulling="2026-05-25 11:54:11.308601167 +0000 UTC m=+2210.063577944" lastFinishedPulling="2026-05-25 11:54:17.355496827 +0000 UTC m=+2216.110473603" observedRunningTime="2026-05-25 11:54:17.975981782 +0000 UTC m=+2216.730958579" watchObservedRunningTime="2026-05-25 11:54:17.978007571 +0000 UTC m=+2216.732984369"
May 25 11:54:26.438216 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:26.438187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:27.442293 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:27.442265    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:28.376406 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:28.376366    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:29.296342 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:29.296309    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:30.224697 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:30.224667    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:31.172250 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:31.172210    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:32.112592 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:32.112560    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:33.041827 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:33.041799    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:33.972688 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:33.972658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:34.889147 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:34.889114    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:35.805654 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:35.805619    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:36.751278 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:36.751244    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:37.740314 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:37.740283    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:38.750778 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:38.750746    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-ggcgj_0a16e53e-5908-4378-bf39-c8858e3a82c9/istio-proxy/0.log"
May 25 11:54:39.848325 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:39.848300    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8_12d991fe-1987-486c-a052-7702a46b9555/istio-proxy/0.log"
May 25 11:54:40.681341 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:40.681312    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8_12d991fe-1987-486c-a052-7702a46b9555/istio-proxy/0.log"
May 25 11:54:41.551263 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:41.551222    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-wqt9m_41b76363-18eb-4414-aab5-1a922915fcc9/manager/0.log"
May 25 11:54:41.559491 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:41.559468    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-lsvdj_82c229ca-664f-4254-80de-5946e4940ae6/limitador/0.log"
May 25 11:54:41.574700 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:41.574673    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-4fgrj_73d1ac66-893f-4b04-9ab9-91f39e4f902f/manager/0.log"
May 25 11:54:43.066635 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.066602    2566 generic.go:358] "Generic (PLEG): container finished" podID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerID="b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9" exitCode=0
May 25 11:54:43.067044 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.066687    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l4z78/must-gather-cfmj8" event={"ID":"e0165b7c-fb5d-4956-adfa-348a67f93f8b","Type":"ContainerDied","Data":"b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9"}
May 25 11:54:43.067044 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.067010    2566 scope.go:117] "RemoveContainer" containerID="b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9"
May 25 11:54:43.235869 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.235845    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-l4z78_must-gather-cfmj8_e0165b7c-fb5d-4956-adfa-348a67f93f8b/gather/0.log"
May 25 11:54:43.882230 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.882195    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qcmnf/must-gather-ks5mb"]
May 25 11:54:43.886923 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.886906    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:43.889143 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.889122    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmnf\"/\"kube-root-ca.crt\""
May 25 11:54:43.889945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.889927    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmnf\"/\"openshift-service-ca.crt\""
May 25 11:54:43.890038 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.889945    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qcmnf\"/\"default-dockercfg-22mcc\""
May 25 11:54:43.894443 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.894418    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmnf/must-gather-ks5mb"]
May 25 11:54:43.978945 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.978897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbjcz\" (UniqueName: \"kubernetes.io/projected/7adaa5c8-a744-4757-a73e-10924363d324-kube-api-access-xbjcz\") pod \"must-gather-ks5mb\" (UID: \"7adaa5c8-a744-4757-a73e-10924363d324\") " pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:43.979119 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:43.979044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7adaa5c8-a744-4757-a73e-10924363d324-must-gather-output\") pod \"must-gather-ks5mb\" (UID: \"7adaa5c8-a744-4757-a73e-10924363d324\") " pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:44.079892 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:44.079850    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7adaa5c8-a744-4757-a73e-10924363d324-must-gather-output\") pod \"must-gather-ks5mb\" (UID: \"7adaa5c8-a744-4757-a73e-10924363d324\") " pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:44.079892 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:44.079902    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbjcz\" (UniqueName: \"kubernetes.io/projected/7adaa5c8-a744-4757-a73e-10924363d324-kube-api-access-xbjcz\") pod \"must-gather-ks5mb\" (UID: \"7adaa5c8-a744-4757-a73e-10924363d324\") " pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:44.080299 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:44.080177    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7adaa5c8-a744-4757-a73e-10924363d324-must-gather-output\") pod \"must-gather-ks5mb\" (UID: \"7adaa5c8-a744-4757-a73e-10924363d324\") " pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:44.088228 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:44.088202    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbjcz\" (UniqueName: \"kubernetes.io/projected/7adaa5c8-a744-4757-a73e-10924363d324-kube-api-access-xbjcz\") pod \"must-gather-ks5mb\" (UID: \"7adaa5c8-a744-4757-a73e-10924363d324\") " pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:44.196140 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:44.196058    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmnf/must-gather-ks5mb"
May 25 11:54:44.331580 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:44.331526    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmnf/must-gather-ks5mb"]
May 25 11:54:44.333258 ip-10-0-143-48 kubenswrapper[2566]: W0525 11:54:44.333229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7adaa5c8_a744_4757_a73e_10924363d324.slice/crio-95a58aa20365caebcd7dd8fb8f899bbf4ccc67e8945640821e7b193649d553c3 WatchSource:0}: Error finding container 95a58aa20365caebcd7dd8fb8f899bbf4ccc67e8945640821e7b193649d553c3: Status 404 returned error can't find the container with id 95a58aa20365caebcd7dd8fb8f899bbf4ccc67e8945640821e7b193649d553c3
May 25 11:54:45.075265 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:45.075226    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmnf/must-gather-ks5mb" event={"ID":"7adaa5c8-a744-4757-a73e-10924363d324","Type":"ContainerStarted","Data":"95a58aa20365caebcd7dd8fb8f899bbf4ccc67e8945640821e7b193649d553c3"}
May 25 11:54:46.081490 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:46.081454    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmnf/must-gather-ks5mb" event={"ID":"7adaa5c8-a744-4757-a73e-10924363d324","Type":"ContainerStarted","Data":"4749aaa79cd95c53c3735edb99dc951db102d201793725b5161cd6709fc7e753"}
May 25 11:54:46.081490 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:46.081489    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmnf/must-gather-ks5mb" event={"ID":"7adaa5c8-a744-4757-a73e-10924363d324","Type":"ContainerStarted","Data":"573f4cf060110fd9eab9b9f56059cc16e69adcc58126130e4f7cc552496bb410"}
May 25 11:54:46.099023 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:46.098961    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-qcmnf/must-gather-ks5mb" podStartSLOduration=2.284975905 podStartE2EDuration="3.098943125s" podCreationTimestamp="2026-05-25 11:54:43 +0000 UTC" firstStartedPulling="2026-05-25 11:54:44.335349598 +0000 UTC m=+2243.090326372" lastFinishedPulling="2026-05-25 11:54:45.149316818 +0000 UTC m=+2243.904293592" observedRunningTime="2026-05-25 11:54:46.095917088 +0000 UTC m=+2244.850893883" watchObservedRunningTime="2026-05-25 11:54:46.098943125 +0000 UTC m=+2244.853919922"
May 25 11:54:46.703594 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:46.703534    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8smhx_6275e60c-94fe-4622-a186-d02cea5494d0/global-pull-secret-syncer/0.log"
May 25 11:54:46.800529 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:46.800462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6mk6l_9442f5d3-3e19-44ec-8db5-1706c9b11ba2/konnectivity-agent/0.log"
May 25 11:54:46.909380 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:46.909344    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-48.ec2.internal_324b37d0baef0cf5218e4c8fcb924549/haproxy/0.log"
May 25 11:54:48.721199 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:48.721164    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-l4z78/must-gather-cfmj8"]
May 25 11:54:48.721917 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:48.721449    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-l4z78/must-gather-cfmj8" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerName="copy" containerID="cri-o://91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239" gracePeriod=2
May 25 11:54:48.723512 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:48.723455    2566 status_manager.go:895] "Failed to get status for pod" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" pod="openshift-must-gather-l4z78/must-gather-cfmj8" err="pods \"must-gather-cfmj8\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-l4z78\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object"
May 25 11:54:48.724837 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:48.724809    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-l4z78/must-gather-cfmj8"]
May 25 11:54:49.056313 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.056091    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-l4z78_must-gather-cfmj8_e0165b7c-fb5d-4956-adfa-348a67f93f8b/copy/0.log"
May 25 11:54:49.057006 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.056704    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:49.058682 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.058642    2566 status_manager.go:895] "Failed to get status for pod" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" pod="openshift-must-gather-l4z78/must-gather-cfmj8" err="pods \"must-gather-cfmj8\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-l4z78\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object"
May 25 11:54:49.095639 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.095532    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-l4z78_must-gather-cfmj8_e0165b7c-fb5d-4956-adfa-348a67f93f8b/copy/0.log"
May 25 11:54:49.096898 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.096240    2566 generic.go:358] "Generic (PLEG): container finished" podID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerID="91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239" exitCode=143
May 25 11:54:49.096898 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.096410    2566 scope.go:117] "RemoveContainer" containerID="91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239"
May 25 11:54:49.096898 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.096576    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l4z78/must-gather-cfmj8"
May 25 11:54:49.098662 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.098610    2566 status_manager.go:895] "Failed to get status for pod" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" pod="openshift-must-gather-l4z78/must-gather-cfmj8" err="pods \"must-gather-cfmj8\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-l4z78\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object"
May 25 11:54:49.116452 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.116101    2566 scope.go:117] "RemoveContainer" containerID="b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9"
May 25 11:54:49.129164 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.128320    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh889\" (UniqueName: \"kubernetes.io/projected/e0165b7c-fb5d-4956-adfa-348a67f93f8b-kube-api-access-fh889\") pod \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") "
May 25 11:54:49.129164 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.128407    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0165b7c-fb5d-4956-adfa-348a67f93f8b-must-gather-output\") pod \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\" (UID: \"e0165b7c-fb5d-4956-adfa-348a67f93f8b\") "
May 25 11:54:49.145358 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.145047    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0165b7c-fb5d-4956-adfa-348a67f93f8b-kube-api-access-fh889" (OuterVolumeSpecName: "kube-api-access-fh889") pod "e0165b7c-fb5d-4956-adfa-348a67f93f8b" (UID: "e0165b7c-fb5d-4956-adfa-348a67f93f8b"). InnerVolumeSpecName "kube-api-access-fh889". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:54:49.151641 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.151610    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0165b7c-fb5d-4956-adfa-348a67f93f8b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e0165b7c-fb5d-4956-adfa-348a67f93f8b" (UID: "e0165b7c-fb5d-4956-adfa-348a67f93f8b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:54:49.158233 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.157926    2566 scope.go:117] "RemoveContainer" containerID="91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239"
May 25 11:54:49.158642 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:54:49.158434    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239\": container with ID starting with 91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239 not found: ID does not exist" containerID="91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239"
May 25 11:54:49.158642 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.158473    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239"} err="failed to get container status \"91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239\": rpc error: code = NotFound desc = could not find container \"91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239\": container with ID starting with 91c8c8561badd1fb10c41bdbb3ec18f3ff9888151ed39419de9e9e77484fa239 not found: ID does not exist"
May 25 11:54:49.158642 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.158504    2566 scope.go:117] "RemoveContainer" containerID="b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9"
May 25 11:54:49.159138 ip-10-0-143-48 kubenswrapper[2566]: E0525 11:54:49.159069    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9\": container with ID starting with b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9 not found: ID does not exist" containerID="b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9"
May 25 11:54:49.159138 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.159097    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9"} err="failed to get container status \"b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9\": rpc error: code = NotFound desc = could not find container \"b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9\": container with ID starting with b78ddcff1a3555b1f7da971e81bcbd961c2e52d476cd31dbcd076ce37e4105e9 not found: ID does not exist"
May 25 11:54:49.229234 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.229167    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh889\" (UniqueName: \"kubernetes.io/projected/e0165b7c-fb5d-4956-adfa-348a67f93f8b-kube-api-access-fh889\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:54:49.229234 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.229205    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0165b7c-fb5d-4956-adfa-348a67f93f8b-must-gather-output\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
May 25 11:54:49.411692 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.411632    2566 status_manager.go:895] "Failed to get status for pod" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" pod="openshift-must-gather-l4z78/must-gather-cfmj8" err="pods \"must-gather-cfmj8\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-l4z78\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object"
May 25 11:54:49.837896 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:49.837802    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" path="/var/lib/kubelet/pods/e0165b7c-fb5d-4956-adfa-348a67f93f8b/volumes"
May 25 11:54:51.368561 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:51.365191    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-wqt9m_41b76363-18eb-4414-aab5-1a922915fcc9/manager/0.log"
May 25 11:54:51.391720 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:51.391677    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-lsvdj_82c229ca-664f-4254-80de-5946e4940ae6/limitador/0.log"
May 25 11:54:51.423974 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:51.423943    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-4fgrj_73d1ac66-893f-4b04-9ab9-91f39e4f902f/manager/0.log"
May 25 11:54:52.762285 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:52.762252    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ss9q2_99202225-27af-4ec5-8722-ee88f742653b/node-exporter/0.log"
May 25 11:54:52.788004 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:52.787980    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ss9q2_99202225-27af-4ec5-8722-ee88f742653b/kube-rbac-proxy/0.log"
May 25 11:54:52.812323 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:52.812290    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ss9q2_99202225-27af-4ec5-8722-ee88f742653b/init-textfile/0.log"
May 25 11:54:55.432072 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.432039    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"]
May 25 11:54:55.432684 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.432594    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerName="gather"
May 25 11:54:55.432684 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.432614    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerName="gather"
May 25 11:54:55.432684 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.432636    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerName="copy"
May 25 11:54:55.432684 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.432643    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerName="copy"
May 25 11:54:55.432916 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.432727    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerName="copy"
May 25 11:54:55.432916 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.432744    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0165b7c-fb5d-4956-adfa-348a67f93f8b" containerName="gather"
May 25 11:54:55.436955 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.436930    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.444872 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.444849    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"]
May 25 11:54:55.489686 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.489656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-podres\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.489838 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.489695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-sys\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.489838 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.489718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-lib-modules\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.489838 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.489769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzjz\" (UniqueName: \"kubernetes.io/projected/88b3144b-a54e-4c8e-92a7-1e3a37df837f-kube-api-access-9fzjz\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.489838 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.489800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-proc\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.590481 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.590445    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-podres\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.590736 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.590703    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-podres\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.590901 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.590766    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-sys\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.591056 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.591034    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-lib-modules\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.591196 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.591177    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzjz\" (UniqueName: \"kubernetes.io/projected/88b3144b-a54e-4c8e-92a7-1e3a37df837f-kube-api-access-9fzjz\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.591362 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.591336    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-proc\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.591464 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.590827    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-sys\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.591518 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.591496    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-proc\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.591590 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.591533    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88b3144b-a54e-4c8e-92a7-1e3a37df837f-lib-modules\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.599710 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.599668    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzjz\" (UniqueName: \"kubernetes.io/projected/88b3144b-a54e-4c8e-92a7-1e3a37df837f-kube-api-access-9fzjz\") pod \"perf-node-gather-daemonset-jgn7r\" (UID: \"88b3144b-a54e-4c8e-92a7-1e3a37df837f\") " pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.750767 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.750680    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:55.901242 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:55.901213    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"]
May 25 11:54:56.148844 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:56.148807    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r" event={"ID":"88b3144b-a54e-4c8e-92a7-1e3a37df837f","Type":"ContainerStarted","Data":"089d3cdf01058664bf10e298de01bc120a6764c9afe8f2b5a60c83b2d7682040"}
May 25 11:54:56.148844 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:56.148844    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r" event={"ID":"88b3144b-a54e-4c8e-92a7-1e3a37df837f","Type":"ContainerStarted","Data":"c22ba9a8586ced3ae67b50c7af63dc788c28bdaa3b5369d5f89983bb92012a87"}
May 25 11:54:56.149081 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:56.148864    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:54:56.167040 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:56.166997    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r" podStartSLOduration=1.166983479 podStartE2EDuration="1.166983479s" podCreationTimestamp="2026-05-25 11:54:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:54:56.163639625 +0000 UTC m=+2254.918616422" watchObservedRunningTime="2026-05-25 11:54:56.166983479 +0000 UTC m=+2254.921960276"
May 25 11:54:56.810329 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:56.810297    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7gl7_26513b67-e7e4-43c9-8a09-d496226e067e/dns/0.log"
May 25 11:54:56.831353 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:56.831329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7gl7_26513b67-e7e4-43c9-8a09-d496226e067e/kube-rbac-proxy/0.log"
May 25 11:54:56.904262 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:56.904236    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jqzls_7ad17068-4f7d-491f-8d77-61c903d241cb/dns-node-resolver/0.log"
May 25 11:54:57.459556 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:57.459516    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z7trl_d14d47c1-872a-4915-9cc5-f57cf19c52bb/node-ca/0.log"
May 25 11:54:58.433080 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:58.433047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-4g6n8_12d991fe-1987-486c-a052-7702a46b9555/istio-proxy/0.log"
May 25 11:54:58.943592 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:58.943525    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnpqg_608f8a96-f300-4efe-89d2-51ca889a3e67/serve-healthcheck-canary/0.log"
May 25 11:54:59.533944 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:59.533913    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qh96l_b44825ca-8493-4c51-9956-1d2f5137d19f/kube-rbac-proxy/0.log"
May 25 11:54:59.555653 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:59.555631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qh96l_b44825ca-8493-4c51-9956-1d2f5137d19f/exporter/0.log"
May 25 11:54:59.582005 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:54:59.581979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qh96l_b44825ca-8493-4c51-9956-1d2f5137d19f/extractor/0.log"
May 25 11:55:02.162612 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:02.162585    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qcmnf/perf-node-gather-daemonset-jgn7r"
May 25 11:55:02.695077 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:02.695039    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-559cc6dd6f-qpvw7_30f14716-eb7a-44f0-9a9f-c44a2636fbeb/manager/0.log"
May 25 11:55:02.720276 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:02.720251    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84bbbfb696-p4wtr_159887ab-b2ff-4ec4-9571-f363baef9f4d/manager/0.log"
May 25 11:55:02.741209 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:02.741188    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-t5pkt_0a34b29f-05db-4110-ad8e-d49951a1aa73/server/0.log"
May 25 11:55:03.133420 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:03.133385    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-f6h99_65d1e61e-0dfa-41bd-bf39-4387253f6b00/manager/0.log"
May 25 11:55:03.152837 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:03.152814    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rvn65_cdde9eb4-7d47-43f6-9cf5-da674cbc0959/s3-init/0.log"
May 25 11:55:03.185001 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:03.184977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-2t78h_8196380f-10fd-41c7-86e4-87af8dfa4179/seaweedfs/0.log"
May 25 11:55:09.312229 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.312200    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5f87w_e5d18530-e2b7-49bf-906a-2969cfb88e1d/kube-multus-additional-cni-plugins/0.log"
May 25 11:55:09.336442 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.336414    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5f87w_e5d18530-e2b7-49bf-906a-2969cfb88e1d/egress-router-binary-copy/0.log"
May 25 11:55:09.374486 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.374450    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5f87w_e5d18530-e2b7-49bf-906a-2969cfb88e1d/cni-plugins/0.log"
May 25 11:55:09.423860 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.423835    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5f87w_e5d18530-e2b7-49bf-906a-2969cfb88e1d/bond-cni-plugin/0.log"
May 25 11:55:09.454719 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.454695    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5f87w_e5d18530-e2b7-49bf-906a-2969cfb88e1d/routeoverride-cni/0.log"
May 25 11:55:09.478225 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.478205    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5f87w_e5d18530-e2b7-49bf-906a-2969cfb88e1d/whereabouts-cni-bincopy/0.log"
May 25 11:55:09.499750 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.499727    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5f87w_e5d18530-e2b7-49bf-906a-2969cfb88e1d/whereabouts-cni/0.log"
May 25 11:55:09.915226 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:09.915180    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mm888_c4dc4e95-98e6-4e49-95d6-d3ccedf3cdcb/kube-multus/0.log"
May 25 11:55:10.081600 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:10.081531    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xm9qk_61545c61-84b8-432a-9443-e9fd8b59060f/network-metrics-daemon/0.log"
May 25 11:55:10.100006 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:10.099976    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xm9qk_61545c61-84b8-432a-9443-e9fd8b59060f/kube-rbac-proxy/0.log"
May 25 11:55:10.910887 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:10.910818    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/ovn-controller/0.log"
May 25 11:55:10.952835 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:10.952804    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/ovn-acl-logging/0.log"
May 25 11:55:10.983595 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:10.983568    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/kube-rbac-proxy-node/0.log"
May 25 11:55:11.010021 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:11.009995    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/kube-rbac-proxy-ovn-metrics/0.log"
May 25 11:55:11.030066 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:11.030036    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/northd/0.log"
May 25 11:55:11.083752 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:11.083705    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/nbdb/0.log"
May 25 11:55:11.129242 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:11.129215    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/sbdb/0.log"
May 25 11:55:11.319522 ip-10-0-143-48 kubenswrapper[2566]: I0525 11:55:11.319445    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6fxqg_e915550d-e2e5-4bdb-b20c-6e2e2cf9756d/ovnkube-controller/0.log"